Test-Time Steering for Lossless Text Compression via Weighted Product of Experts (2025)

Abstract: Lossless compression techniques are crucial in an era of rapidly growing data. Traditional universal compressors like gzip offer low computational overhead, high speed, and broad applicability across data distributions. However, they often lead to worse compression rates than modern neural compressors, which leverage large-scale training data to model data distributions more effectively.Despite their advantages, neural compressors struggle to generalize to unseen data. To address this limitation, we propose a novel framework that performs Test-Time Steering via a Weighted Product of Experts (wPoE).At inference, our method adaptively combines a universal compression model with a pretrained neural language model, ensuring the compression rate is at least as good as the best individual model.Extensive experiments demonstrate that our approach improves the performance of text compression without requiring fine-tuning. Furthermore, it seamlessly integrates with any autoregressive language model, providing a practical solution for enhancing text compression across diverse data distributions.

## introduction lossless text compression is a long-standing research challenge. various general-purpose compression algorithms, e.g., gzip @xcite and lzma2 @xcite , have been proposed to efficiently encode text data. these algorithms are widely adopted due to their low computational overhead, high efficiency, and consistent performance across diverse data distributions. shannon's source coding theorem @xcite establishes that the lower bound of average code length for a given dataset is determined by its shannon entropy. while general-purpose compression algorithms are practical and efficient, their inability to model the underlying data distribution limits their ability to approach this bound. to overcome this limitation, neural network-based models trained to minimize the @xcite divergence between the data distribution and the model's distribution have emerged as powerful alternatives @xcite @xcite . empirical evidence suggests that neural compression techniques often outperform traditional methods, particularly when the input text closely resembles the data used for training. large language models (llms) @xcite @xcite have further advanced this capability, with studies such as @xcite and @xcite demonstrating that leveraging llms can significantly enhance compression performance. despite these successes, neural compression models often struggle to generalize to unseen data and require intensive computation @xcite . in contrast, classical universal compressors, while not always achieving the highest compression ratios, maintain stable performance across diverse datasets without requiring training @xcite . additionally, these traditional methods typically have lower computational overhead. @xcite ; @xcite frame lossless compression and language modeling as two aspects of the same underlying principle, emphasizing the importance of developing more adaptable language models to improving text compression. to address this limitation, we propose a novel framework within the test-time steering (tts) setting. unlike methods that require fine-tuning on the target domain, tts aims to adapt a pre-trained model to the target distribution, typically with a lightweight cost and a limited number of observed data points. specifically, we introduce a weighted product of experts (wpoe) method that dynamically combines a universal compression model with a pre-trained neural model. by selectively modulating each expert's contribution based on the input text's characteristics, wpoe effectively adapts to shifts in data distribution. in summary, our contributions are as follow: ## related work universal compression lossless data compression is a central topic of information theory. a data sequence is assumed to be generated by a random process. the goal is to design lossless compressors and decompressors that achieve the theoretical limit, i.e., the entropy rate of the data, asymptotically as the length of the data sequence goes to infinity. when the underlying distribution/statistics of data is known, optimal lossless compression can be achieved by methods like huffman coding. however, in most real-world applications, the exact distribution is usually hard to obtain and the data we are given is a single realization of this distribution. this motivates the framework of universal compression, in which we assume the underlying distribution belongs to a known family of distributions and require that the compressor and the decompressor should not be a function of the underlying distribution. the goal of universal compression is to design a single compression scheme that universally achieves the optimal theoretical limit, for every distribution in the family, without knowing which distribution generates the data. over the past five decades, various universal compressors have been developed. for the family of independent and identically distributed sequences, the laplace compressor @xcite and the krichevsky-trofimov (kt) compressor @xcite ) are known to be universal. for the family of stationary ergodic processes, the lempel-ziv compressors @xcite and the burrows-wheeler transform @xcite achieve the optimal performance. for finite memory processes, techniques such as context tree weighting @xcite have been developed. these universal compression techniques have found applications beyond theoretical research, playing a crucial role in standard data compressor such as gzip @xcite , image formats like gif(compuserve, 1987) and tiff @xcite , and file compressors like bzip2 @xcite . while most of the universal compressors can be shown to achieve the entropy rate in the asymptotic limit, the speeds they converge to the theoretical limit are known to be slow, even for variations of these algorithms that are highly optimized and widely adopted in practical file compression. ## background in this section, we explain the equivalence between prediction and compression in more detail and conclude that the problem of lossless compression boils down to the design of a probability distribution for the given data that is as close to the true data distribution as possible. ## method in this section, we propose a novel lossless text compressor that is robust and generalizable. we start by explaining how we design the conditional probabilities p θ (x i |@xmath0. ## experimental evaluation in this section, we evaluate the effectiveness of our framework through text compression experiments on diverse text datasets using various large language models (llms). language models. we evaluate two categories of pretrained language models. following @xcite , the first category consists of byte-level decoder-only transformers of varying sizes-200k, 800k, and 3.2m-trained on the enwik8 dataset @xcite . the second category includes pretrained open-source large language models, such as gpt-2 @xcite and llama-3 @xcite . for comparison, we also report the performance of widely used universal compressors, including gzip @xcite and lzma2 @xcite . datasets. we consider a wide range of text datasets: enwik8, enwik9, code, math, and shakespeare. enwik9 @xcite consists of the first 10 9 bytes of the english wikipedia dump on march 3, 2006. meanwhile, enwik8 is the first one-tenth portion of enwik9. the code dataset, introduced by @xcite , is a code generation benchmark comprising 1,140 fine-grained python programming tasks, spanning 139 libraries across 7 domains, which ensures that the dataset is sufficiently diverse in the coding domain. the math dataset, published by @xcite , contains 12,500 challenging competition mathematics problems spanning seven subjects, e.g., algebra, geometry, and number theory. shakespeare (shakespeare, 2007) contains the complete works, plays, sonnets, and poems of william shakespeare. following the setting of @xcite , we partition all datasets into sequences of 2048 bytes to optimize inference efficiency in transformer-based models, such as gpt-2 and llama-3. ## conclusion in this work, we introduced a novel test-time steering framework for lossless text compression based on the weighted product of experts (wpoe). by integrating a pretrained autoregressive language model with a training-free universal compressor, i.e., a naive bayes classifier with laplace smoothing, our method effectively enhances outof-distribution performance without the need for additional retraining. the wpoe framework guarantees that the integrated model performs at least as well as its best individual expert. our extensive experiments across multiple datasets demonstrate that our approach consistently improves compression rates, particularly on ood data where neural compressors typically struggle. notably, we observed that even with a very limited amount of data, our optimization procedure converges to a near-optimal weight within just 10 iterations when using a second-order optimizer. moreover, by progressively incorporating additional experts, we further boost performance, with the inclusion of the naive bayes classifier contributing significantly to compression improvements. these findings underscore the potential of the weighted product of experts (wpoe). for future work, we believe that incorporating other suitable experts could further enhance generalization across various nlp tasks. more specifically, if an effective prior can be identified for a given task, integrating it with a pretrained model could improve performance with minimal additional cost.