Paper: Language Modeling Is Compression (ICLR-2024)

Reviewer comment (Evaluation):
While its LLM compressor performs well compared to gzip, it is difficult to use in practice due to the high inference cost.

Author response:
While we agree that viewing language modeling as compression is a well-established insight (see, e.g., Section 1), we believe that it is valuable to the broader ML research community to popularize these ideas (beyond the LLM professionals) and the subsequent connections to essential areas, such as in-context learning, scaling laws, tokenization, and generative modeling. Our goal is thus not to provide new insight in terms of compression methodology or to advocate for using LLMs as compressors in practice, as they have too many limitations (e.g., size and inference time) to be practical. Instead, we aim to provide an interesting experimental evaluation of viewing language modeling through the lens of compression.

Reviewer comment (Novelty):
Ge et al. (2023) proposed using lora tuning to adapt the LLM for the compression ability, enabling it to compress a long context into a short span. Although Ge et al.'s (2023) work is not lossless compression, they observed similar phenomena: for example, their Table 3 comparison of normal text, patterned random text, and completely random text shows that LLMs can compress based on certain patterns (even though the model has not seen patterned random text, it performs better on patterned random text than on completely random text). Similarly, their Tables 5 and 8 also indicate that more potent LLMs are better at compression. Therefore, while the perspective of this paper is novel and interesting, its final conclusions cannot be considered entirely novel.

Author response:
We thank the reviewer for pointing us to the paper by Ge et al. (2023), which we have added to our manuscript. The findings of Ge et al. (2023) are akin to our investigation of tokenization, which can be viewed as a pre-compression of the input to a language model. That is, Ge et al. (2023) present an alternative to tokenization that solves the practical issue of maximizing the information in the context window of a Transformer, which boosts downstream performance.

However, Ge et al. (2023) do not show that their ICAE method achieves strong lossless compression rates on image and audio data, and the performance on patterned random text is relatively poor (3.5 BLEU) compared to normal text (99.3 BLEU) and almost the same as completely random text (0.2 BLEU). In contrast, we show that LLMs are on-par or even *outperforming specialized image and audio compressors* (in terms of raw compression rate, i.e., not taking into account the model-size or program complexity).

Moreover, unlike Ge et al. (2023), we not only show that larger LLMs achieve better compression rates but conduct a comprehensive investigation of the scaling behavior across model *and* dataset size, elucidating the limits of naive scaling.

Thus, as the reviewer pointed out, we still believe that a major novelty of our work is showing that *language* models trained on text are strong general-purpose compressors on image and audio data and *comprehensively* investigating the scaling laws for compression. We want to emphasize again that the ability for general-purpose compression implies that LLMs, through training on a large corpus of text, reliably acquire the ability to detect and predict (and thus compress) general algorithmic patterns. This is not trivially expected, and while understanding the precise nature of these patterns and how they relate to the training data remains an exciting avenue for future research, the finding that competitive compression rates on images and audio data can be achieved is not trivially expected per se.

Reviewer comment (Evaluation):
If we discuss the number of parameters in larger language models and how it reflects compression performance, it would be better to investigate the reasons behind this relationship.

Author response:
We thank the reviewer for their time and constructive feedback on the paper.


**What is the relationship between the number of parameters and compression performance?**

We investigate the relationship between the number of parameters and compression performance in Figure 2, where we evaluate the adjusted compression rate (i.e., taking the model size into account) on datasets of increasing sizes ($10^7$, $10^8$, and $10^9$ bytes). First, we note that for a fixed dataset size, the compression rate shows a distinctive U-shape, i.e., it improves with model size but starts to deteriorate when the model becomes too big (i.e., overfits). However, we also see that the compression performance improves with model size as the dataset size improves since the compression rate improves faster than the parameter count (i.e., the models show evidence of emergent capabilities with scale).

Reviewer comment (Presentation):
A key into understanding the algorithm is Fig. 1, but the figure contains ambiguities and confusion. E.g. Why a "?" in "b0?" only? In the last column, how to go from 4bit to 7bit? The illustration should be more tractable.

Author response:
**Can you clarify Figure 1?**

Yes, we have extended the caption of the figure to further clarify arithmetic coding and included the step-by-step example below in Appendix A.

Figure 1. visualizes arithmetic encoding, which iteratively partitions the interval $I = [0, 1)$ according to a predictive model $P$ and an input string, i.e., $AIXI$ for Figure 1.

First, we construct the intervals for $P(\cdot)$:
* $[0, 0.45)$ for $P(A) = 0.45$
* $[0.45, 0.75)$ for $P(I) = 0.3$
* $[0.75, 1)$ for $P(X) = 0.25$

Since the first token is $A$, we set $I = [0, 0.45)$ and iterate. Thus, the intervals for $P(\cdot | A)$ are:

* $[0.2 * 0, 0.2 * 0.45) = [0, 0.09)$ for $P(A | A) = 0.2$
* $[0.09, (0.2 + 0.6) * 0.45) = [0.09, 0.36)$ for $P(I | A) = 0.3$
* $[0.36, (0.2 + 0.6 + 0.2) * 0.45) = [0.36, 0.45)$ for $P(X | A) = 0.2$

Since the next token is $I$, we set $I = [0, 0.45)$ and so on. We terminate with $I = [0.322, 0341)$ for $AIXI$. Next, arithmetic encoding will compute the binary sequence corresponding to iteratively splitting the interval $[0, 1)$ in half until it is fully contained in $I$. 

Concretely, this yields the binary sequence.

* $b0$ -> $[0, 0.5)$
* $b01$ -> $[0.25, 0.5)$ 
* $b010$ -> $[0.25, 0.375)$
* $b0101$ -> $[0.3125, 0.375)$
* $b01010$ -> $[0.3125, 0.34375)$
* $b010101$ -> $[0.328125, 0.34375)$
* $b0101010$ -> $[0.328125, 0.3359375)$

As $[0.328125, 0.3359375)$ is fully contained in $I = [0.322, 0341)$, the compressed output is $0101010$, which consists of 7 bits as opposed to the 4 bytes used to encode $AIXI$.

Reviewer comment (Novelty):
I am also doubtful about the part on tokenization. The tokenizer being part of a Transformer is only due to its root in language modeling. In the same vein, we can easily make a claim on "A CNN stem is compression" (a CNN has a stem, body & head), and varying the stem (e.g. different strides) we get different compression rates too, so what is interesting in that?

Author response:
**What is the motivation for the section on tokenization?**

The motivation for the tokenization section is precisely what the reviewer points out: Tokenization is rooted in language modeling and has been developed to overcome practical limitations (e.g., small context windows). Here, we describe that a tokenizer can be viewed as an additional compressor that is applied before the language model. Note that we do not state that "Transformers Are Compression" but that "Language Modeling Is Compression". In that sense, a CNN trained to predict the next token with the cross-entropy loss will also yield a lossless compressor. Here, we only consider Transformers since they are predominantly used for language modeling research.

Reviewer comment (Experiments):
Is the context length of 2048 bytes still a must given the recent work on lifting this length constraint? e.g. "Efficient Streaming Language Models with Attention Sinks".

Author response:
**Why do you use a context length of only 2048 bytes?**

While the reviewer correctly points out that recent work has enabled much longer contexts, we are limited to a context size of 2048 bytes due to the computational resources available to us. However, we do not expect the results to change qualitatively, given that Figure 4 shows that the compression rates plateau after a certain context length.

Reviewer comment (Writing):
The motivation of this work is rather unclear to me. Is this work about advocating the use of pre-trained large language models as a potential method for compression? If so, how can they be used as such in practice considering their limitations? Or is it about using the compression framework to better understand large language models? If so, why is it interesting to study pre-trained large language models "through the lens of compression"?

Author response:
**What is the motivation for this work and what are the implications of the results?**

In recent years, language modeling has become a central element of AI research, primarily thanks to transformers' unmatched efficiency and performance (particularly for large foundation models). However, many researchers are unaware of the close connection between prediction and compression power. Thus, our primary motivation is to demonstrate that well-known large language models can be used to compress data. One, arguably even more surprising, finding is that models trained to predict/compress text turn out to become fairly good compressors for data that they haven't been trained on (i.e., image and audio data). This "transfer" can only be explained by LLMs obtaining the ability to predict, and thus compress, general (algorithmic) patterns. As we show further, larger LLMs perform better at general compression than smaller LLMs, suggesting that the ability for predicting more general patterns improves with scale and is thus obtained via training on text (rather than an inherent capability of untrained LLMs). Moreover, we show that studying foundation models through the lens of lossless compression sheds light on various essential aspects of language modeling:
* ***In-context learning***: Compressing longer sequences is easier given the more available information (i.e., few-shot learning or meta-learning).
* ***Tokenization***: In this view, a tokenizer is just an extra (manually conceived) compression step.
* ***Generation***: One can sample from a compressor using a softmax over the lengths of the compressed data. We use this to empirically demonstrate the relation between stronger compressors and higher-quality generations.
* ***Scaling laws***: Model size affects the downstream compression performance.
Finally, we show that, surprisingly, *language* models are strong compressors for image and audio data, which provides evidence for the claim that LLMs learn to compress more general (algorithmic) patterns with scale (Mirchandani et al., 2023).

However, as the reviewer points out correctly, we do not advocate for using LLMs as compressors in practice, as they have too many limitations (e.g., size and inference time) to be practical.  Rather, we advocate for using compression as an insightful tool for the evaluation of LLMs, and the compression-viewpoint as an insightful, complementary, theoretical angle.

Suvir Mirchandani et al. Large Language Models as General Pattern Machines. https://arxiv.org/abs/2307.04721.

Reviewer comment (Novelty):
The authors mention that they “advocate for using (lossless) compression to study foundation models”. Why and what benefits does this framework have? It is not clear to me how the results in this paper should help my understanding of large language models beyond their use as compressors? What are the further implications of the results?

Author response:
**What is the motivation for this work and what are the implications of the results?**

In recent years, language modeling has become a central element of AI research, primarily thanks to transformers' unmatched efficiency and performance (particularly for large foundation models). However, many researchers are unaware of the close connection between prediction and compression power. Thus, our primary motivation is to demonstrate that well-known large language models can be used to compress data. One, arguably even more surprising, finding is that models trained to predict/compress text turn out to become fairly good compressors for data that they haven't been trained on (i.e., image and audio data). This "transfer" can only be explained by LLMs obtaining the ability to predict, and thus compress, general (algorithmic) patterns. As we show further, larger LLMs perform better at general compression than smaller LLMs, suggesting that the ability for predicting more general patterns improves with scale and is thus obtained via training on text (rather than an inherent capability of untrained LLMs). Moreover, we show that studying foundation models through the lens of lossless compression sheds light on various essential aspects of language modeling:
* ***In-context learning***: Compressing longer sequences is easier given the more available information (i.e., few-shot learning or meta-learning).
* ***Tokenization***: In this view, a tokenizer is just an extra (manually conceived) compression step.
* ***Generation***: One can sample from a compressor using a softmax over the lengths of the compressed data. We use this to empirically demonstrate the relation between stronger compressors and higher-quality generations.
* ***Scaling laws***: Model size affects the downstream compression performance.
Finally, we show that, surprisingly, *language* models are strong compressors for image and audio data, which provides evidence for the claim that LLMs learn to compress more general (algorithmic) patterns with scale (Mirchandani et al., 2023).

However, as the reviewer points out correctly, we do not advocate for using LLMs as compressors in practice, as they have too many limitations (e.g., size and inference time) to be practical.  Rather, we advocate for using compression as an insightful tool for the evaluation of LLMs, and the compression-viewpoint as an insightful, complementary, theoretical angle.

Suvir Mirchandani et al. Large Language Models as General Pattern Machines. https://arxiv.org/abs/2307.04721.

Reviewer comment (Experiments):
No experiments with pre-trained models other than Chinchilla-70B. Having more models could provide more evidence on the compression capabilities of pre-trained large language models and to see how compression capabilities correlate with prediction performance.

Author response:
**Can you add experiments with publicly available pre-trained models other than Chinchilla-70B?**

Yes, we added an experimental evaluation of the compression capabilities of LLaMA1-7B to Table 1. LLaMA1-7B performs comparably to the Chinchilla models, except for ImageNet data, where it achieves the best compression (most likely since it has seen images as part of the CommonCrawl dump, which makes up 67% of its training data).

Moreover, we have open-sourced the code for reproducibility and will link it to the paper upon acceptance.

Reviewer comment (Reproducibility):
Not using publicly available pre-trained large language models for reproducibility.

Author response:
**Can you add experiments with publicly available pre-trained models other than Chinchilla-70B?**

Yes, we added an experimental evaluation of the compression capabilities of LLaMA1-7B to Table 1. LLaMA1-7B performs comparably to the Chinchilla models, except for ImageNet data, where it achieves the best compression (most likely since it has seen images as part of the CommonCrawl dump, which makes up 67% of its training data).

Moreover, we have open-sourced the code for reproducibility and will link it to the paper upon acceptance.