Revisiting Block-based Quantisation: What is Important for Sub-8-bit LLM Inference? (2023)

Abstract: The inference of Large language models (LLMs) requires immense computation and memory resources. To curtail these costs, quantisation has emerged as a promising solution, but existing LLM quantisation mainly focuses on 8-bit. In this work, we explore the statistical and learning properties of the LLM layer and attribute the bottleneck of LLM quantisation to numerical scaling offsets. To address this, we adapt block quantisations for LLMs, a family of methods that share scaling factors across packed numbers. Block quantisations efficiently reduce the numerical scaling offsets solely from an arithmetic perspective, without additional treatments in the computational path. Our nearly-lossless quantised 6-bit LLMs achieve a 19\times higher arithmetic density and 5\times memory density than the float32 baseline, surpassing the prior art 8-bit quantisation by 2.5\times in arithmetic density and 1.2\times in memory density, without requiring any data calibration or re-training. We also share our insights into sub-8-bit LLM quantisation, including the mismatch between activation and weight distributions, optimal fine-tuning strategies, and a lower quantisation granularity inherent in the statistical properties of LLMs. The latter two tricks enable nearly-lossless 4-bit LLMs on downstream tasks. Our code is open-sourced.

## introduction pre-trained large language models (llms) @xcite @xcite have demonstrated impressive performance on a range of natural language processing (nlp) tasks. however, their underlying computational and memory costs are a critical bottleneck to their usability. for instance, the larger variants in the gpt family scale up to hundreds of billions of parameters, requiring at least 300gb of memory to store 1 https://github.com/chengzhang-98/llm-mixed-q these parameters in a float16 format @xcite . quantisation serves as a natural solution for reducing the cost of running inference on these llms @xcite @xcite , as a low-precision format enables cost savings across all relevant efficiency metrics: reduced on-chip memory, increased arithmetic intensity for matrix multiplies, and decreased dram bandwidth requirement. on the other hand, the growing popularity of running services such as chatgpt @xcite provides an impetus for exploring the use of custom silicon to support llm inference. this raises the question: what would a low-precision number system look like in these near-future llm hardware accelerators (asics)? llm quantisation is challenging because of the activations with large absolute magnitudes, also known as activation outliers @xcite . previous approaches have proposed various techniques to address such outliers. however, these either require additional treatments in the integer quantisation domain (llm.int8() and smoothquant) or yield unsatisfactory performance (zeroquant); and prior work has primarily focused on arithmetics that can be ported to gpus. we observe that the presence of outliers necessitates different scaling factors at a finer granularity than per-tensor or per-token level @xcite . this insight naturally leads us to revisit arithmetic systems with small exponents, such as minifloat @xcite , block minifloat @xcite , block logarithm @xcite , and block floating point @xcite , as they can effectively represent outliers in transformer models. to the best of our knowledge, our work is the first to systemically investigate shortexponent arithmetics for llm quantisation. figure 1 illustrates the variance of the tensors joining the gemms in an opt-6.7b @xcite ⃝ and 5 ⃝ in float32/float16 format, which take up 20.6% floating-point operations in opt-6.7b's self-attention. * means outliers in llm.int8() is computed in float16; this improves arithmetic density but memory density is kept identical to canonical float16. ## related work while quantisation of earlier machine learning (ml) models has been extensively studied, effective quantisation of llms still remains an open problem. in this section, we review the previous works on block-based quantisation and compare to the existing llm quantisation techniques. ## method in this section, we outline our quantisation strategy for llms. we first define block-based quantisation and then describe the metrics we use for evaluating quantisation methods. finally, we detail a precision search that lowers the quantisation granularity down to the tensor level, effectively accommodating the statistical distribution inherent in llms. ## evaluation we conducted a comprehensive set of experiments to identify the key factors influencing the performance of sub-8-bit llms. we begin with a language modelling task to eliminate less promising quantisation methods (section 4.2), and then run the promising ones on downstream tasks. for the tasks that proved challenging even for fp32 models, we resort to fine-tuning. additionally, we conducted a mixed-precision search on two tasks where the quantised 4-bit model struggle. the results of this search provide insights into how to further refine quantisation at the tensor level. ## conclusion this study focuses on addressing the scaling offset issue in llms and provides valuable insights into the quantisation of llms. through extensive experimentation, we identify key factors that significantly impact llm quantisation. when aiming for quantisation above or equal to 6-bit, bfp surpasses previous methods in terms of accuracy, memory density, and arithmetic density, without requiring for data calibration or training. moreover, we demonstrate that fine-tuning or mixed precision techniques enable 4-bit llms on downstream tasks. fine-tuning is suitable for gpus, and mixed precision has the potential to shift the inference platform from gpus to cost-effective asics. our findings contribute to advancing the field of llm quantisation and provide practical guidance for achieving good quantisation performance. our work has been open-sourced and will also contribute to an ml hardware framework named mase for hardware deployment @xcite .