Text Compression for Efficient Language Generation (2025)

Abstract: We challenge the prevailing assumption that LLMs must rely fully on sub-word tokens for high-quality text generation. To this end, we propose the “Generative Pretrained Thoughtformer” (GPTHF), a hierarchical transformer language model capable of text generation by compressing text into sentence embeddings and employing a sentence attention mechanism. GPTHF retains GPT’s architecture, modifying only token interactions via dynamic sparse attention masks. Our experiments show that GPTHF achieves an up to an order of magnitude improvement in FLOPs efficiency and a threefold increase in runtime speed compared to equally-sized GPT models in the low-size regime. This is achieved through a unique generation method that caches and reuses sentence embeddings, allowing significant portions of the input to bypass large parts of the network.

## introduction the development of llms has garnered substantial interest due to their impressive capabilities in nlp tasks. the dominant paradigm for improving llms has been scaling, with models scaling from hundreds of millions (e.g. bert, @xcite ) to over a trillion parameters (e.g. switch transformer, @xcite ) in a span of four years. while these massive scales unlock remarkable performance across nlp tasks @xcite , they come with substantial costs in hardware, energy, and time @xcite , requiring the exploration for more efficient methods. efforts to improve efficiency include pruning @xcite , quantization @xcite , and knowledge distillation @xcite . mixture of experts models @xcite further reduced inference costs while preserving capacity. however, one area remains under-explored: the reliance of llms on sub-word tokens, each requiring embeddings several kilobytes in size. this raises the question of whether more condensed text representations could offer similar performance with greater efficiency. models like the funnel-transformer @xcite hint at potential gains through compressing and subsequently decompressing hidden states. going one step further, we introduce gpthf, a hierarchical transformer that compresses entire sentences into fixed-size embeddings. we explore whether such representations still carry sufficient semantic payload to maintain generation quality, thereby asking if sub-word tokens could possibly be eliminated for greater computational efficiency. experimental results show that gpthf achieves strong perplexity scores, follows scaling laws in the low-parameter regime, and operates at a significantly reduced flops cost and inference time. ## related work a new line of research explored the idea of a "hierarchical transformer," a transformer operating on variable-size embeddings within different layers of the network. early examples include the models of @xcite and @xcite . the funnel transformer @xcite compressed token sequences via incremental pooling, with inter-layer skip connections allowing later layers to access pre-compressed information. when re-investing the saved flops, the funnel transformer outperformed previous state-of-the-art models with comparable computational resources. @xcite expanded this idea to generative transformers with their "hourglass" model, demonstrating improved perplexity on a wikipedia dataset. other examples include sentence-bert @xcite and sentence-gpt @xcite , focus on generating sentence embeddings for downstream tasks. our work differs from all of the above in several ways. instead of compressing a fixed-size group of tokens, we compress a sentence -a unit of higher semantic value in language -into one embedding. we focus on leveraging these embeddings to improve computational efficiency, not on the embeddings themselves. ## limitations a central question remains in whether transformers can generate high-quality text using only compressed sentence embeddings with sufficient size and training. while smaller gpthf models follow scaling laws similar to gpts, their inability to reliably finish sentences highlights challenges tied to either scale or the compression method itself. further training on larger models is necessary to determine if this limitation is inherent to compression or surmountable via scaling. future work should evaluate these models on downstream tasks to assess practical utility beyond perplexity. additionally, integrating gpthf with existing optimizations like kv-caching could yield better speedups, though diminishing returns are a potential challenge. comprehensive ablation studies focusing on key parameters like hidden size could offer deeper insights into performance. alternative approaches, such as directly generating sentence embeddings and subsequently decompressing, warrant exploration to enhance or complement current methods.