C omp A ct: Compressed Activations for Memory-Efficient LLM Training (2025)

Abstract: We introduce CompAct, a technique that reduces peak memory utilization on GPU by 25-30% for pretraining and 50% for fine-tuning of LLMs. Peak device memory is a major limiting factor in training LLMs, with various recent works aiming to reduce model memory. However most works don’t target the largest component of allocated memory during training: the model’s compute graph, which is stored for the backward pass. By storing low-rank, compressed activations to be used in the backward pass we greatly reduce the required memory, unlike previous methods which only reduce optimizer overheads or the number of trained parameters. Our compression uses random projection matrices, thus avoiding additional memory overheads. Comparisons with previous techniques for either pretraining or fine-tuning show that CompAct substantially improves existing compute-performance tradeoffs. We expect CompAct’s savings to scale even higher for larger models.

## introduction training large language models (llms) and finetuning them on downstream tasks has led to impressive results across various natural language applications @xcite . however, as llms scale from millions to hundreds of billions of parameters, the computational resources required for both pre-training and fine-tuning become prohibitive. while compute power is the primary bottleneck for those who train very large llms, memory requirements become the main limitation for researchers without access to vast hardware resources. this disparity severely limits the ability to advance the field of llm training to only a select few. recent works tackle memory reductions by applying a low rank approximation to model parameters @xcite , or to * equal contribution. gradients after the backward pass @xcite @xcite . however, as seen in figure 1 , the main memory component is the computation graph itself. its size also scales with batch size, in contrast with other memory components. in this work, we introduce compact, a novel optimization technique that saves low-rankcompressed activations during the forward pass, instead of the full activation tensors. consequently, the resulting gradients are low-rank as well, also reducing the size of optimizer states. as compact decompresses the gradients back to full size only for the update step, it compresses a large part of the compute graph, which in turn translates to major memory savings. figure 2 presents an overview of our method, while overall, compact achieves savings of about 17.3% of peak device memory with practical batch sizes, for pretraining llama-350m @xcite , and 50% for fine-tuning roberta-base @xcite . as seen in figure 1 , the estimated memory reduction achieved by compact increases with model size. for llama-65b, the estimated reduction is approximately 30%. for llama-350m, the estimated reduction is around 21%, which is 4% higher than the observed empirical value. this suggests a realistic memory saving range of 25%-30% for llama-65b. choosing the low-rank projection used for com-pression is critical, as it can impact performance and reduce training throughput. by using a random projection matrix sampled on the fly as in @xcite , we eliminate the cost of computing and storing optimal projection matrices, which can be slow to compute and large enough to reduce our memory savings. we show sound theoretical motivation justifying the use of random projections from recent works in random sketching theory @xcite . finally, we present experimental measurements demonstrating that compact reduces memory significantly with minimal impact on model performance for both pretraining and finetuning. there are various approaches to making the training process more efficient, including mixed precision training @xcite , quantization methods @xcite @xcite ; anonymous, 2024a), parallelism techniques @xcite @xcite , and low rank approximation methods. the latter being mostly unexplored for pretraining, it is the focus of this work. ## experiments in this section, we evaluate compact on both pretraining (section 4.1) and finetuning tasks (section 4.2). in all experiments, we apply compact to all attention and mlp blocks in all layers of the model, except for the output projection in the attention mechanism. for further details, see appendix b. moreover, we provide a comparison of com-pact's throughput and memory usage with other methods in section 4.3, and explore various types of projection matrices in section 5. ## ablation study this section presents a series of ablation experiments examining how different design choices and hyperparameters affect training performance, memory usage, and convergence. first, we explore the effects of different projection matrices on training performance when applied within the compact framework. we evaluate the following projection matrices: gaussian projection this is our primary method, where each layer samples a different gaussian random matrix. gaussian projection with shared seed by setting the same random seed for all layers, we sample identical projection matrices for all layers (where dimensions permit). this investigates whether sharing the same subspace among different layers influences learning performance. ## conclusion in this work, we presented compact, a memoryefficient method for training llms by compressing activations, gradients, and optimizer states of linear layers. we demonstrate that compact achieves significant memory savings for training llms, reaching 25%-30% memory reduction for pretraining llama-65b and 50% for roberta-base, with minimal impact on training throughput and performance. our method is easily scalable, applicable to various model sizes, and is easily composable with other techniques. by directly compressing the compute graph during training, compact targets a major component of peak device memory which was neglected in recent works. we believe this approach should guide future work for further memory gains. a good example could be incorporating sparse random projections into compact, which would reduce the computational cost associated with sampling and matrix operations. another area for improvement is the approximation of intermediate activations, such as those generated by flashattention, hadamard products, and non-linearities, which require significant memory. by addressing these memory-intensive operations, compact's memory reductions can be extended even further.