QUIK : Towards End-to-end 4-Bit Inference on Generative Large Language Models (2024)

Abstract: Large Language Models (LLMs) from the GPT family have become extremely popular, leading to a race towards reducing their inference costs to allow for efficient local computation. However, the vast majority of existing work focuses on weight-only quantization, which can reduce runtime costs in the memory-bound one-token-at-a-time generative setting, but does not address costs in compute-bound scenarios, such as batched inference or prompt processing.In this paper, we address the general quantization problem, where both weights and activations should be quantized, which leads to computational improvements in general. We show that the majority of inference computations for large generative models can be performed with both weights and activations being cast to 4 bits, while at the same time maintaining good accuracy. We achieve this via a hybrid quantization strategy called QUIK that compresses most of the weights and activations to 4-bit, while keeping a small fraction of “outlier” weights and activations in higher-precision. QUIK is that it is designed with computational efficiency in mind: we provide GPU kernels matching the QUIK format with highly-efficient layer-wise runtimes, which lead to practical end-to-end throughput improvements of up to 3.4x relative to FP16 execution. We provide detailed studies for models from the OPT, LLaMA-2 and Falcon families, as well as a first instance of accurate inference using quantization plus 2:4 sparsity.Anonymized code is available.

## introduction large language models (llms) from the generative pretrained transformer (gpt) family @xcite are extremely popular. one surprising property is the ability to quantize them, e.g., @xcite @xcite , enabling efficient local generative inference for these models, even on personal computers. the vast majority of work on llm quantization can be categorized as follows: thus, there is still a significant gap between compressed formats supported by hardwarespecifically, nvidia gpus natively support accelerated 4bit matrix multiplication on both the ampere and lovelace architectures (nvidia, 2023)and quantization algorithms which would allow accurate inference on hardware-supported formats. contribution. in this paper, we look to bridge this gap, and show that a large fraction of the computation in modern llms such as opt @xcite , llama-2 @xcite and falcon (tii uae, 2023) can be performed accurately and efficiently using 4-bit activations and weights (4w4a). on the algorithmic side, we show significantly improved results relative to prior work on joint quantization of weights and activations to 4 bits, via a hybrid scheme for quantization to int4 with gpu kernel support, called quik. in quik, matrices are split into "base" weights and activations, which are processed exclusively at 4-bit precision, and a small number of "outlier" weights and activations, which are processed at higher precision such as int8 or fp16. using this approach, as well as additional insights into layer sensitivity, we build a 2.61x 3.4x 2.48x speedup figure 1 : accuracy and speedups for quik at different model sizes, on the llama family of models. quik achieves up to 3.4x speedup with minor accuracy degradation on llama-2 models. framework which can recover accuracy within 0.3-0.5 perplexity points across model sizes, while executing a large fraction of the inference in int4. for illustration, for the sensitive llama2 model with 70b parameters, we can recover accuracy within 0.5 perplexity, while executing 70% of the linear layer computations in int4, leading to 3.4x endto-end speedups (see figure 1 ). we consider our work orthogonal to kv-cache quantization. on the performance side, the key feature of quik is that it can be implemented efficiently via gpu kernels with low runtime and memory overheads relative to gpu-native int4 matrix multiplication (matmul). we demonstrate this via a general implementation leading to per-layer speedups and end-to-end throughput improvements relative to both fp16 and int8 baselines. specifically, we show that supporting a limited number of feature and weight outliers can have negligible overhead by fusing the quantization and dequantization operations into the matmul and by mitigating their costs in linear layers via additional optimizations. overall, quik leverages quantization for significant end-to-end speedups and memory reductions. for example, for processing a sequence of 2048 tokens on a commodity rtx 3090 gpu, we achieve end-to-end speedups between 3.1x, for the opt-66b and falcon-180b models, and 3.4x for llama2-70b, relative to a theoretical optimum of ≈4x. in addition, quik requires much less gpu memory, and therefore, less gpus, relative to fp16. for instance, quik provides 3.6x memory reduction for opt-66b, and 3x compression for accurate execution of llama2-70b, executing the latter in less than 50gb of gpu memory. figure 2 clearly shows that the case of few tokens (1 and 16) the operation is bound by memory transfer, whereas the it becomes compute-bound for token counts larger than 64-128. a realistic endto-end llm deployment would need to consider optimizing both scenarios, as the prompt processing "prefill" case falls into the large token count scenario, whereas generating one-token-at-a-time falls into the former case. moreover, running a "batched" version of the single-token workload, i.e. for multiple users, would again result in large token counts, returning to the compute-bound case. notice that existing methods for weight-only quantization @xcite @xcite only reducing the amount of data which needs to be transferred per operation, but still perform the computation in the original precision. thus, they do not help in the compute-bound case, and in fact even slightly increase the amount of computation per operation, due to the de-quantization overheads. ## experimental validation general setup. we evaluate our method on opt @xcite , llama-2 @xcite , and falcon (tii uae, 2023) models, using huggingface @xcite implementations of model definitions and datasets. following smoothquant @xcite , we extract outlier indices using 512 random sentences from the pile dataset @xcite . we consider up to 5% (based on the model size) of the input features as outliers in the linear layers. during the gptq weight quantization, we randomly select 128 samples with 2048 sequence length from the c4 dataset @xcite . we apply symmetric quantization to weights and asymmetric quantization to activations. clipping thresholds for weight quantization are found via a linear search over the squared error. quik quantizes a 70b model in less than 2h on a single a100 gpu. ## limitations our current experiments are limited to compressing the linear layers of llms. however, our scheme is compatible with virtually any scheme for compressing attention layers or the kv-cache @xcite , which can be applied orthogonally. another limitation, which we plan to address in future work, is experimenting with recent mixture-of-experts (moe) architectures, and integration with speculative decoding @xcite .