P rocrustes GPT : Compressing LLM s with Structured Matrices and Orthogonal Transformations (2025)

Abstract: Large language models (LLMs) demonstrate impressive results in natural language processing tasks but require a significant amount of computational and memory resources. Structured matrix representations are a promising way for reducing the number of parameters of these models. However, it seems unrealistic to expect that weight matrices of pretrained models can be accurately represented by structured matrices without any fine-tuning.To overcome this issue, we utilize the fact that LLM output is invariant under certain orthogonal transformations of weight matrices.This insight can be leveraged to identify transformations that significantly improve the compressibility of weights within structured classes.The proposed approach is applicable to various types of structured matrices that support efficient projection operations. Code is available at: https://github.com/GrishKate/ProcrustesGPT.

## introduction large language models have achieved remarkable success in language processing and are widely used in a variety of applications, but their deployment is still challenging, as these models hardly fit into a single gpu and require much computational resources during the inference and training processes. the research community is actively seeking efficient algorithms to reduce the size of pretrained models without sacrificing accuracy. one approach that has not been fully explored is the use of structured matrices, which can potentially not only reduce the number of parameters in the model but also speed up computations. lowparametric matrix decomposition can be applied directly to the weight matrices, minimizing the difference between the original and decomposed weights, i.e., ∥w -w ′ ∥ → min w ′ ∈s , where s is the low-parametric matrix class. such a factorization may appear to be a reasonable initial guess for further fine-tuning. unfortunately, it often leads to high approximation errors for any s when no additional training is performed, as there are no restrictions on the structure of the weight matrices during pretraining. to overcome this issue, we utilize invariance of the network output under certain orthogonal transformations, which was first observed in @xcite . in particular, for each layer, we aim to find such transformations that lead to the best compressibility within the chosen matrix class s. in our paper, we focus on the sum of kronecker product representation and gs-matrices @xcite , but other representations are also possible. finding an optimal orthogonal transformation is a known linear algebra problem and is called the orthogonal procrustes problem. the resulting framework is formulated as an optimization problem on the weights of the pretrained network, is free from the need for fine-tuning and applicable for different structured matrix representations. in figure 1 , we present relative errors in the frobenius norm for different layers with and without using orthogonal transformations. we observe a noticeable increase in compression ability thanks to optimally chosen orthogonal transformations. the main contributions of our work: ## related work existing approaches to neural network compression can be divided into four categories: quantization, distillation, pruning, and matrix factorization. matrix factorization is a promising and relatively underexplored technique, and the most widely used approach within this category is a low-rank approximation. the work @xcite has shown that by carefully selecting ranks for each of the weights, the performance of llms can be improved on some tasks. however, direct application of svd to uniform compression of weights leads to poor performance because weights usually have high ranks @xcite . instead, the authors @xcite @xcite @xcite use calibration datasets or fisher information matrix to approximate activations. other works explore low-rank plus sparse @xcite and low-rank plus quantized decompositions @xcite . the works @xcite have been among the first to apply the kronecker-product decomposition for the compression of bert and gpt2. the paper @xcite proposes to enhance the performance of the kronecker decomposition by adding permutations. these works compress the weight matrices directly and require knowledge distillation or fine-tuning in order to recover performance. the authors of modegpt @xcite propose to split the weight matrices of the transformer into pairs and jointly compress them using svd, cr or nyström approximation. this method is training-free and efficiently preserves model performance; however, it does not address compression of embedding and head matrices. many studies have investigated embedding compression, but most of the proposed algorithms require additional training. the works @xcite apply the tensor train decomposition to the embedding layer, which offers strong compression, but requires training from scratch or hinders model performance. other methods @xcite approximate embedding with low-rank decomposition, but these methods require fine-tuning. groupreduce @xcite utilizes knowledge about words occurrence by weighing tokens with their frequencies and applies block low-rank approximation. our goal in this study is to maintain model performance after weight factorization without additional training. the authors of the prunning approach @xcite have introduced the concept of computational invariance, meaning that orthogonal transformations can be applied to transformer weights without changing the output of the model. slicegpt uses invariance to project layer activations onto their principal components and remove columns or rows from weight matrices. in this work, we utilize computational invariance to find a better approximation of the weights with structured representations. ## our approach our approach consists in iteratively finding orthogonal transformations q to obtain optimal compression properties of the matrix weights. for brevity, we call these orthogonal transformations "rotations", although they do not necessarily have determinants equal to one. in section 3.1, we present the concept of rotational invariance. then, in section 3.2, we present our approach as an optimization formulation to be solved. notably, @xcite appears to be a particular instance of this formulation, which we also discuss further in section 4.3. ## structured matrix representations in this section, we present different structured matrix representations on which we focus in our work. the sum of kronecker products yields the most consistent accuracy gain within different models. although the gs-matrix representation resulted in slightly lower accuracy overall, its computationally efficient structure offers significant potential to accelerate inference. finally, we examine matrices with zero blocks, revealing connections with pruning techniques (ashkboos et al., 2024). ## experiments 6.1 setup we implement our method for opt @xcite and llama2 @xcite ) models using hugging face transformers @xcite . as the calibration data we use 128 sequences of length 2048 from wikitext2 dataset @xcite . experiments were run on a single v100 gpu with 32gb memory or a100 gpu with 80gb memory. we evaluate zero-shot performance using lm evaluation harness @xcite across arc-e, arc-c @xcite , piqa @xcite , winogrande @xcite , and hellaswag @xcite datasets. ## conclusion this paper presents an approach to llm compression with structured matrix factorizations, which is suitable for compression with various types of decompositions, including kronecker products and gs matrices. our method maintains performance in generation and zero-shot tasks, and does not require recovery fine-tuning. we hope this work will inspire further research on training-free compression with structured representations.