CHESS : Optimizing LLM Inference via Channel-Wise Thresholding and Selective Sparsification (2024)

Abstract: Deploying large language models (LLMs) on edge devices presents significant challenges due to the substantial computational overhead and memory requirements. Activation sparsification can mitigate these resource challenges by reducing the number of activated neurons during inference. Existing methods typically employ thresholding-based sparsification based on the statistics of activation tensors. However, they do not model the impact of activation sparsification on performance, resulting in suboptimal performance degradation. To address the limitations, this paper reformulates the activation sparsification problem to explicitly capture the relationship between activation sparsity and model performance. Then, this paper proposes CHESS, a general activation sparsification approach via CHannel-wise thrEsholding and Selective Sparsification. First, channel-wise thresholding assigns a unique threshold to each activation channel in the feed-forward network (FFN) layers. Then, selective sparsification involves applying thresholding-based activation sparsification to specific layers within the attention modules. Finally, we detail the implementation of sparse kernels to accelerate LLM inference. Experimental results demonstrate that the proposed CHESS achieves lower performance degradation over eight downstream tasks while activating fewer parameters than existing methods, thus speeding up the LLM inference by up to 1.27x.

## introduction large language models (llms) have become integral to diverse applications, including code generation, office assistance, voice assistance, and assistive applications for individuals with disabilities. however, due to the substantial computation and memory requirements of llm inferences, deploying llms on edge devices is still challenging. to * corresponding author mitigate these overheads, utilizing the inherent activation sparsity of llm has emerged as a promising strategy @xcite @xcite . this approach has proven effective for models with the relu activation function @xcite . contemporary llms demonstrate that swiglu or geglu activation functions can further boost the model performance, but they induce less activation sparsity. consequently, several methods @xcite are proposed to increase sparsity by applying regularization to the activation function and employing continuous training. however, those works require fine-tuning the llms, which entails significant overheads. to avoid these overheads and increase activation sparsity in modern llms, @xcite propose a thresholding-based pruning method to actively sparsify the activation tensors during the inference stage. however, this thresholding technique focuses solely on the statistics of the activation tensors themselves, failing to model the impact of sparsification on overall model performance, which results in suboptimal performance degradation. to address the above limitations, this paper proposes chess , a new activation sparsification optimization via channel-wise thresholding and selective sparsification. first, this paper reformulates the activation sparsification problem to explicitly capture the relationship between activation sparsity and model performance. then, this paper proposes channel-wise thresholding for ffn modules in llms, which determines a unique threshold for each activation channel. furthermore, this paper proposes selective sparsification, which applies thresholding-based activation sparsification to the target layers in the attention module. finally, this paper presents the implementations of sparse kernels to accelerate the inference based on the sparse activations. to validate the effectiveness of the proposed chess , this paper conducts comprehensive experiments on various downstream tasks and stateof-the-art llms. experimental results demonstrate that chess can achieve lower performance degradation with better end-to-end inference speedup. the code is publicly available foot_0 . ## experiments in this section, this paper first introduces the dataset, comparisons, and implementation details. then, this paper presents the main results over 8 downstream tasks in terms of the model performance and model efficiency. besides, this paper also conducts an ablation study across different sparsification methods for the attention module and analysis on performance and efficiency over different sparsity level. additionally, this paper conducts extended comparisons with other state-of-the-art training-free pruning methods, to validate the effectiveness of the proposed chess . ## related work various methods have been proposed to address the challenges associated with deploying llms locally. weight quantization @xcite @xcite aims to represent llm weights using lower bit-widths, thereby reducing memory usage and access overhead. activation quantization focuses on minimizing the memory footprint of activation tensors and kv cache @xcite @xcite . these methods can be applied along with our proposed chess method. weight pruning @xcite involves setting a portion of the llm weights to zero to reduce computational overhead and memory requirement. however, this approach faces several challenges including noticeable degradation in performance and limited hardware support when applied on personal devices. non-autoregressive decoding approaches, such as speculative decoding @xcite or medusa @xcite , seek to convert autoregressive decoding process of llms into parallel decoding to mitigate memory access overhead. however, these methods impose increased computational demands, which presents significant challenges for deployment on personal devices with limited processing capabilities. ## conclusion this paper reformulates the activation sparsification problem and introduces the chess , a general activation sparsification via channel-wise thresholding and selective sparsification. experiments show that the proposed chess can achieve a lower performance degradation and accelerate the llm inference with sparse activations.