When Compression Meets Model Compression: Memory-Efficient Double Compression for Large Language Models (2024)

Abstract: Large language models (LLMs) exhibit excellent performance in various tasks. However, the memory requirements of LLMs present a great challenge when deploying on memory-limited devices, even for quantized LLMs. This paper introduces a framework to compress LLM after quantization further, achieving about 2.2x compression ratio. A compression-aware quantization is first proposed to enhance model weight compressibility by re-scaling the model parameters before quantization, followed by a pruning method to improve further. Upon this, we notice that decompression can be a bottleneck during practical scenarios. We then give a detailed analysis of the trade-off between memory usage and latency brought by the proposed method. A speed-adaptive method is proposed to overcome it. The experimental results show inference with the compressed model can achieve a 40% reduction in memory size with negligible loss in accuracy and inference speed.

## introduction large language models (llms) exhibit excellent performance in various natural language tasks and have made significant advancements in recent years @xcite @xcite . however, llms with billion-scale parameters present a high demand for memory @xcite . deploying these llms on memory-limited devices poses a huge challenge due to their extensive memory requirements. the model size of large language models has been developing at a fast pace @xcite . the memory or gpu memory is far from sufficient to run llms, even for lightweight models. model compression is one promising approach to address this challenge, referring to the methods that reduce model size @xcite . quantization is one of the model compression techniques widely used for compressing llms. it approximates the model's weights with shorter bits to reduce the model size. nonetheless, the int8 quantized 7b model (a relatively small large-language model) still needs over 7gb of memory. a substantial gap remains due to the conflict between large model parameter sizes and insufficient memory. this paper aims to reduce the memory gap by further compressing the quantized models. recent work demonstrates that there's still compressibility in quantized llm models @xcite . following this, we first employ a lossless compression algorithm @xcite to explore the compressibility of quantized model data. to investigate and improve the compressibility, we analyze the data distribution before and after quantization. the quantized data with higher compressibility presents two key observations that can be utilized for further enhanced compressibility: 1) data with uneven distribution exhibits higher compressibility; 2) the quantized data have a higher proportion of nearzero values. these observations suggest that modifying the data distribution can improve data compressibility. simply applying a scaling technique to model weights makes the quantized weight distribution more uneven, which can improve compressibility but reduce accuracy by approximately 60%. to address this issue, we propose a compression-aware quantization and pruning approach that expands important values and reduces unimportant weight. although the compressed model significantly reduces memory usage for llms, the frequent decompression operation may also introduce overhead during the model inference process. to alleviate this problem, we first implement a compressed model inference framework and analyze overall inference speed based on system memory architecture. we then propose a speed-adaptive method to address the bottleneck caused by decompression overhead. by partially compressing the model data, we can improve the total decompress throughput. additionally, we provide strategies to balance memory usage and inference speed. the experimental results show a remarkable compression ratio (cr), about 2.2, in quantized model size. importantly, this cr is achieved with minimal compromise on model accuracy within a 1% drop. inference with compressed llm can reduce 40% memory size without affecting the speed. our approaches present a promising solution to the memory challenges faced by llms, enabling their deployment on memory-limited devices. ## explore the compressibility of llms we first examine the compressibility characteristics of model data, focusing on model weights and activations. our analysis reveals that temporary activations achieve a high compression ratio, whereas weights exhibit a low compression ratio. we then explore enhancing weight compressibility through scaling operations. compressibility of llms. int8 quantized model can be further compressed. to investigate this compressibility, we employ a general compression algorithm zstd @xcite to compress the quantized activations/weights data from three models of varying sizes. the compressibility is quantified using the compression ratio (cr), calculated by dividing the data size before compression by the size after compression. compressing weights presents more challenges than compressing activations. however, optimizing weight compression is crucial for efficient memory use since it accounts for most of the memory overhead. as shown in ## conclusion we propose a novel double compression method for llms, which combines model compression with lossless compression algorithms. motivated by the observation that weight data in llms exhibit compressibility and the potential for further improvement in compression ratio, we present several approaches to enhance the compressibility. we also investigate the decompression overhead of inference with compressed models and propose an adaptive method to overcome the overhead. our work provides a solution to reduce memory size and facilitate the deployment of llms. 16981 this work's limitation mainly lies in the quantization methods for llms. we focus on compressing int8 quantized weights, as int8 quantization is the most widely used method and is supported by lots of hardware. the model compressibility of quantized models may vary with different methods.