Saten: Sparse Augmented Tensor Networks for Post-Training Compression of Large Language Models (2025)

Abstract: The efficient implementation of large language models (LLMs) is crucial for deployment on resource-constrained devices. Low-rank tensor compression techniques, such as tensor-train (TT) networks, have been widely studied for over-parameterized neural networks. However, their applications to compress pre-trained LLMs for downstream tasks (post-training) remains challenging due to the high-rank nature of pre-trained LLMs and the lack of access to pretraining data. In this study, we investigate low-rank tensorized LLMs during fine-tuning and propose sparse augmented tensor networks (Saten) to enhance their performance. The proposed Saten framework enables full model compression. Experimental results demonstrate that Saten enhances both accuracy and compression efficiency in tensorized language models, achieving state-of-the-art performance.

## introduction transformers have shown great success in various natural language processing tasks @xcite @xcite . however, their large number of parameters and computational demands hinder their implementation on resource-constrained devices. consequently, various methods for llm compression have been studied, including pruning @xcite , distillation @xcite , quantization @xcite , and matrix factorization @xcite @xcite . low-rank tensor factorization is one of the prominent neural network compression techniques that has been widely studied and applied to various neural network architectures @xcite @xcite . the tensor-train (tt) format is the most common tensor decomposition method used for neural network 1 representative code and implementation details are available at: https://github.com/rmsolgi/saten.git . despite their success in improving the efficiency of llm fine-tuning and pre-training @xcite , low-rank tensor compression of pretrained language models has not yet been successfully reported. this task mainly faces two challenges. first, almost all open-source llms are trained using uncompressed approaches, and the resulting model parameters of many layers may not exhibit a low-rank structure. second, there is a lack of access to pre-training data to fully retrain low-rank parameters. these challenges result in a significant performance drop when post-training compression of language models is performed during fine-tuning using low-rank tensor factorization. ## background tensors are generalizations of matrices to higher orders and can be represented by multi-dimensional data arrays @xcite . a real tensor of order d and dimension can be denoted as @xmath0, where n j is the size of dimension j. the (i obviously, matrices and vectors are tensors of order @xmath1, respectively. tensor train (tt) decomposition. many practical tensors have low ranks and can be approximated using low-rank tensor decompositions. the tt decomposition approximates an order-d tensor a with d factors {g @xmath2: here, operator × 1 3 refers to contracting the third and first dimensions of the left and right tensors, respectively (see appendix a for more details). ## the saten method in this section, we present the saten framework to compress a pre-trained llm. since many layers of a pre-trained llm may not exhibit a low-rank property, compressing a pre-trained model directly using tt decomposition can result in huge accuracy drop. to address this challenge, saten approximates the matrix w as follows: the decomposition in saten is illustrated in fig. 1 . ## experimental results in this section, we conduct experiments to compress bert-base @xcite and llama-3.2-1b (ai, 2024). for details regarding hyperparameter for fine-tuning and saten compression, see appendix d. in appendix c, we also show additional results regarding the compression of distil-bert @xcite , with a particular focus on (1) the effectiveness of saten on the embedding layer, (2) the balance between sparsity and lowrank parameterization. in the following, we focus on discussing the overall performance on bert-base and llama-3.2-1b. ## limitations we demonstrated that saten reduces the theoretical computational time complexities (i.e. macs) compared to the uncompressed model. current mainstream platforms for implementing large language models are not optimized for tt plus sparse operations. nevertheless, there is ongoing interest and progress in improving sparse operations for ai and machine learning tasks @xcite . meanwhile, in this study, we utilized structured sparsity, which facilitates the implementation of saten on gpus @xcite and kept the unstructured sparsity at high levels of 95% for which sparse algebra packages typically achieve significant speedup. our findings motivate future research on custom hardware accelerators as the actual inference speedup of saten requires both software-and hardware-level optimization, which is the subject of our future studies. another promising direction for future work is the application of quantization to further enhance compression efficiency and reduce inference cost @xcite . finally, we evaluated the llama 3.2 models, which are already distilled and thus more difficult to compress. this highlights saten's practical utility in resource-constrained, low-redundancy settings. although we do not include further largerscale models, saten's architecture-agnostic design and layer-wise compression make it scalable, and we expect the observed improvements to generalize across model sizes and architectures.