Revisiting Offline Compression: Going Beyond Factorization-based Methods for Transformer Language Models (2023)

Abstract: Recent transformer language models achieve outstanding results in many natural language processing (NLP) tasks. However, their enormous size often makes them impractical on memory-constrained devices, requiring practitioners to compress them to smaller networks. In this paper, we explore offline compression methods, meaning computationally-cheap approaches that do not require further fine-tuning of the compressed model. We challenge the classical matrix factorization methods by proposing a novel, better-performing autoencoder-based framework. We perform a comprehensive ablation study of our approach, examining its different aspects over a diverse set of evaluation settings. Moreover, we show that enabling collaboration between modules across layers by compressing certain modules together positively impacts the final model performance. Experiments on various NLP tasks demonstrate that our approach significantly outperforms commonly used factorization-based offline compression methods.

## introduction the recent trend of pre-training transformer @xcite language models on enormous unsupervised corpus has led to outstanding performances on many downstream tasks. for downstream tasks, these pre-trained models are then either fine-tuned @xcite @xcite or the prompting paradigm @xcite is used (especially in the so-called large language models), which avoids having a different model per task @xcite @xcite . in each of the two paradigms, it has been shown that increasing the scale of language models generally leads to better performance on a range of downstream tasks @xcite . indeed, for autoregressive language models, @xcite demonstrated a power-law relationship between the number of parameters and the respective performance. @xcite further showed that certain abilities of language models emerge only when the number of its parameters passes certain thresholds, providing an incentive to scale these models further. although scaling up these language models make them empirically powerful across many diverse tasks, it makes them infeasible to train for many nlp practitioners due to huge pre-training costs. more importantly, even using the available pre-trained models for inference is becoming more challenging (especially for memory-constrained applications like edge devices), with recent models having hundreds of billions of parameters @xcite . with the rise of nlp model sizes, there have been many efforts to compress transformer language models without compromising their performance. although being inherently different, many of these efforts rely on knowledge distillation @xcite to help the compressed model (i.e., the student model) better imitate the parent model (i.e., the teacher model). however, these approaches often need costly distillation on upstream or downstream tasks @xcite as well as expensive data augmentation techniques @xcite to help improve the compressed model performance. these approaches become even less feasible when enormous language models are being distilled. another line of research focuses on computationally-cheap methods (i.e., offline compression) where a smaller model can be achieved from a pre-trained model without it being necessarily fine-tuned over a downstream or upstream task. these offline methods include weight pruning @xcite , weight quantization @xcite , tensor factorization @xcite @xcite and hybrid approaches @xcite . this paper proposes a novel offline factorizationbased method for compressing transformer language models. the paper's main goal is to propose an offline method that produces a competitive language model (compared to the original model perplexity) before any fine-tuning is performed. similar to @xcite , we use an autoencoder model (see figure 1a ) to compress different modules' weights. however, unlike the previous work, our approach is not limited to the token embeddings (a.k.a. word embeddings) and can be applied to other transformer modules as well. we also propose and thoroughly investigate the impact of various enhancements for the approach of obtaining the compressed model. it is worth noting that although the experiments and ablation studies are only done bert base model, our in section 4 we demonstrate that applying small changes to the autoencoder architecture (e.g., introducing non-linearity to the decoder) and its loss objective results in superior performance to the singular value decomposition (svd) baseline as measured by model perplexity and its performance on the downstream tasks. moreover, inspired by the redundancies present across self-attention heads @xcite , in section 4.2 we show that the compressed models perform in general better when compressing certain modules from different layers together. additionally, in section 4.6 we investigate the effectiveness of a (parameter) sensitivity-based 2 compression by incorporating fisher information (pascanu and bengio, 2013) in the loss objective. we later show that incorporating these weights significantly improve the compressed language model performance (i.e., perplexity). finally, in section 4.7 and in section 4.8 we discuss the performance of our approach in comparison to various offline-compression baselines and demonstrate that our method provides the best or competitive quality of the compressed model. our main contribution can be summarized as 2 we call it sensitivity as it measures how sensitive the model performance is to the reconstruction error of a certain parameter. deep transformer language models have gained increasing attention in recent years since the seminal work of @xcite . many recent efforts demonstrate that scaling up these language models' parameters generally results in better performance on a range of downstream tasks @xcite @xcite . this empirical observation resulted in recent language models having over a thousand times more parameters @xcite @xcite than the bert base model @xcite . although empirically powerful, these models are becoming harder to use for memoryconstrained applications, which led to many efforts toward language model compression in recent literature. although many recent efforts for language model compression take advantage of distillation @xcite techniques to better imitate the uncompressed model (i.e., the teacher model) behavior, this paper focuses mainly on offline compression methods. by offline, we refer to approaches that do not need fine-tuning the whole model on a downstream/upstream dataset. in the case of language model compression, these methods aim to output a compressed model without losing too much performance (measured by perplexity) that can then be fine-tuned (with or without distillation) or prompted for a certain downstream task. it is worth noting that these methods can still be combined with distillation techniques, but starting the finetuning from a better compressed language model would generally reduce its costs of training (e.g., by improving convergence time). offline compression methods, while being diverse, can be roughly categorized into few paradigms, namely weight pruning @xcite @xcite @xcite , quantization @xcite @xcite , tensor factorization @xcite @xcite @xcite and hybrid approaches @xcite . this paper primarily focuses on the effectiveness of low-rank factorization-based approaches in recent literature @xcite @xcite as an offline compression method. @xcite proposed a svd-based @xcite technique to compress the token embedding module. later works have shown that more complex architectures like autoencoders can result in better compression quality than svd methods @xcite . by taking advantage of the autoencoder, we are able to more easily enforce different properties by either changing its training objective or architecture (e.g., preserving l 2 norm in reconstructed embeddings). @xcite emphasized on the importance of direction in token embedding compression, and in this work we demonstrate its potential importance for other transformer modules as well in different compression ratios. moreover, @xcite showed the significance of redundant information in selfattention heads and compressed different heads (in a certain layer) together to improve compression performance. following a similar idea, we later show that compressing heads from different layers together would generally further boost the compression quality. @xcite proposed a weighted svd (using fisher information) @xcite to outperform the classical svd. we further investigate the benefits of a non-uniform compression (i.e., a weighted reconstruction loss in the autoencoder loss objective) in section 4.6 by analyzing different weighting schemes for parameters. moreover, @xcite proposed using tensor decomposition techniques to compress language models to relatively high compression ratios while using a two-stage distillation technique. moreover, @xcite proposes using the kronecker product as an alternative for the factorization of transformer modules. appendix a.7 discusses using tucker @xcite or kronecker-based methods as an offline approach. it is worth noting that models with relatively high compression ratios become highly dependent on distillation techniques to perform reasonably on downstream tasks. for instance, @xcite claims that even randomly initializing the compressed bert nearly achieves identical performance compared to tensor decomposition from a pre-trained model. ## model our offline compression approach is based on the an autoencoder neural network architecture, similar to @xcite and @xcite . however, in this work, we focus on compressing all the transformer weight matrices rather than just the token embedding matrix. furthermore, we are exploring many more compression improvements using autoencoder as well as investigating architecture-independent techniques. figure 1 : a high-level view of the matrix compression approaches using classical matrix-factorization and autoencoder model that is leveraged in this work. the purpose of the compression is to provide a parameterefficient substitution module to replace the original matrix in the considered model. the autoencoder architecture consists of encoder function h(•) that maps model input @xmath0. the second element of the architecture is the decoder function g(•) responsible for mapping @xmath1. let us assume that we want to compress the matrix @xmath2. using the gradient-descent algorithm, we train the autoencoder model to produce the appropriate approximation ã of the original matrix. as a compressed module, we understand the hidden representation h ψ (a) ∈ r (n×k) together with the decoding module g φ (•). in this setting, the formula for the compression ratio of the original module can be expressed as: our approach to offline compression based on the autoencoder offers flexibility in performing the ablation study as we are able to easily modify its elements, for example, decoder module complexity level or the loss function components. by using an autoencoder architecture with a linear decoder and the rmse cost function, we can obtain the equivalent approximation as provided by a simple matrix factorization. an illustrative comparison of our compression method with the classical matrix factorization approach is shown in figure 1 . following @xcite , we train our autoencoder with the multi-objective cost function consisting of l 2 norm loss and cosine distance loss: ## experiments this section describes our motivations and the results of various analyses and experiments that we conducted to investigate the topic of offline compression thoroughly. we performed our experiments for different weight matrices in the transformer architecture, as each type of weight matrix may have different characteristics, and a given compression method may or may not be appropriate. the analyses described below are performed for token embedding, selfattention (keys, queries, values), and output-dense weight matrices. we focus our study on the bert base model @xcite , but the same methods could be applied to other transformer architectures. all experiments are conducted for three compression ratios (3, 10, and 25) to investigate the differences given the different number of available parameters. all experimental settings of the various studies presented in the following sections are included in appendix a. we evaluate the quality of our compressed models on the masked @xcite language modeling task (using the wikitext-103 test dataset @xcite ) and multiple datasets from the glue benchmark @xcite . ## conclusions this work comprehensively studies various methods for the offline compression of transformer language models. we analyze various changes in the proposed architecture and its optimization function. we test different input modifications and evaluate the compressed language model performance in each scenario. by analyzing various compression settings, we show that our autoencoder-based approach outperforms classical matrix factorization on various nlp downstream tasks. furthermore, we believe the techniques analyzed in this study might also be useful for low-cost compression of different weight matrices unrelated to language models.