M obile Q uant: Mobile-friendly Quantization for On-device Language Models (2024)

Abstract: Large language models (LLMs) have revolutionized language processing, delivering outstanding results across multiple applications. However, deploying LLMs on edge devices poses several challenges with respect to memory, energy, and compute costs, limiting their widespread use in devices such as mobile phones. A promising solution is to reduce the number of bits used to represent weights and activations. While existing works have found partial success at quantizing LLMs to lower bitwidths, e.g. 4-bit weights, quantizing activations beyond 16 bits often leads to large computational overheads due to poor on-device quantization support, or a considerable accuracy drop. Yet, 8-bit activations are very attractive for on-device deployment as they would enable LLMs to fully exploit mobile-friendly hardware, e.g. Neural Processing Units (NPUs). In this work, we make a first attempt to facilitate the on-device deployment of LLMs using integer-only quantization. We first investigate the limitations of existing quantization methods for on-device deployment, with a special focus on activation quantization. We then address these limitations by introducing a simple post-training quantization method, named MobileQuant, that extends previous weight equivalent transformation works by jointly optimizing the weight transformation and activation range parameters in an end-to-end manner. MobileQuant demonstrates superior capabilities over existing methods by 1) achieving near-lossless quantization on a wide range of LLM benchmarks, 2) reducing latency and energy consumption by 20%-50% compared to current on-device quantization strategies, 3) requiring limited compute budget, 4) being compatible with mobile-friendly compute units, e.g. NPU.

## introduction large language models (llms) have markedly advanced language processing capabilities, paving the way for expansive applications in artificial intelligence. however, the deployment of llms is costly in terms of memory, computation, and energy, which can be prohibitive on edge devices like mobile phones. a standard approach to facilitate running these models on edge devices is to quantize them, representing weights and activations with fewer bits, thereby mitigating these costs. existing llm quantization works can be grouped into two categories: weight-only quantization and weight-activation quantization. weightonly quantization approaches @xcite convert model weights into lowbitwidth integers, most commonly 4-bit, and maintain the activations in 16-bit floating-point. weightonly quantization often preserves accuracy while significantly reducing the model storage footprint. in addition, weight-only quantization can result in minor gains in inference latency due to the reduction in memory access overheads. however, these approaches still suffer from high energy consumption and high latency, as computation is performed in floating point. costly on-the-fly weight dequantization is also required during inference. instead, weight-activation quantization approaches forgo the need for on-the-fly dequantization by quantizing both weights and activations, and potentially utilizing efficient fixed-point operators. despite its efficiency benefits, quantizing activations typically degrades accuracy given the activation outliers @xcite @xcite , especially in the case where static per-tensor quantization parameters are applied. to counteract this accuracy drop, previous works include quantizing activations for certain expensive operations @xcite , e.g. matrix multiplication, or employing dynamic per-token quantization @xcite @xcite , which is often slow on graphic processing units (gpus) and, most importantly, lacks hardware support on edge devices. notably, none of these methods support lossless 8bit (int8) per-tensor quantization for the activations, or fully leverage low-precision fixed-point engines, such as the digital signal processor (dsp), or dedicated neural processing unit (npu) @xcite , commonly found in mobile devices @xcite . towards on-device quantization for llms, we introduce mobilequant, a post-training quantization approach that not only effectively handles the conventional accuracy and efficiency challenges of quantization but is also seamlessly supported by existing mobile hardware. to achieve this, mobilequant consists of three simple yet effective methodological extensions, motivated by the shortcomings of existing state-of-theart works when deployed on device, and building on top of these works. these extensions include: 1) applying weight equivalent transformation on all possible layers, 2), learning the optimal quantization range for activations, 3) jointly optimizing all weight transformation and range parameters in an end-to-end manner. as such, mobilequant applies a combination of per-tensor and per-channel weight quantization at 4-bit or 8-bit and per-tensor activation quantization at 8-bit or 16-bit, utilizing fixed-point integer representations for all operations. the benefits of mobilequant over previous works are multifold. firstly, mobilequant enables the quantization of the weights to either 4-bit or 8-bit and the activations to 8-bit integers, except for non-linearities like softmax and normalization, with minimal impact on performance. mobile-quant, hence, maximizes the potential of equivalent transformation-based methods @xcite @xcite that achieve linear-invariant weight equalization. deploying llms on device using mobilequant results in a significant reduction in inference speed and energy usage as the latency and energy consumption of multiply-accumulate operations correlate directly with the bit-widths. besides substantial gains during inference, we also show that mobilequant's end-to-end optimization benefits from more calibration samples and extended training samples through our ablation study. in contrast, previous works that adopt closed-form solutions @xcite , search-based optimization @xcite , and block-wise error minimization @xcite struggle to scale with the number of samples and training steps. lastly, in comparison with other learnable-based quantization methods such as quantization aware training (qat) @xcite , mobilequant retains the model generalizability as the model remains mathematically equivalent to its unquantized variant. our contributions are summarized as follows: ## conclusion we revisited llm quantization from the perspective of deployment on edge devices such as mobile phones. we examined the limitations of current state-of-the-art models for on-device deployment and present mobilequant, the first framework to facilitate compute-, and energy-efficient quantized llms with minimal performance loss. mobile-quant is drop-in compatible with today's edge device hardware and low-level runtimes.