FPE 2 M 2: Approaching Lossless and Efficient Quantization with Native Floating Point (2025)

Abstract: Auto-regressive decoding is a memory-bound job, meaning decoding inference performance is limited by the bandwidth rather than the computational capabilities of the GPU. Weight-only quantization is a promising method to address the memory-bound limitations. Previous studies have followed one of two approaches. Some have exclusively studied integer quantization while ignoring the Gaussian distribution nature of LLMs’ weights. Others have proposed non-uniform quantization but incurred additional I/O overhead due to lookup tables, e.g. NF4. In this work, we extend the IEEE 754 float-point standard to the ExMy quantization schema, which allocates x bit for the exponent and y bit for the mantissa to represent a number. In terms of runtime efficiency, we demonstrate that the conversion from ExMy to FP16 can be realized through register-level operations, which can get almost the same performance as INT5. In terms of quantization loss, we analyze that of different ExMy settings, where the E2M2 schema achieves an optimal balance, offering the highest efficiency with lossless accuracy. We further propose the FPE2M2 framework that supports lossless weight-only quantization inference and validate the FPE2M2 framework on Qwen and LLaMA Models across various modalities, such as text, image, and audio tasks, which achieves a faster inference speed while maintaining nearly lossless accuracy.

## introduction llms have demonstrated outstanding performance and potential for various tasks. however, the high cost of inference has limited their application in practical commercial scenarios. specifically, the pipeline of llm inference can be divided into two stages: pre-filling and decoding. owing to the autoregressive nature of llms, the decoding stage is afflicted by the i/o bottleneck, as it generates only one token at a time. these challenges motivate the research on model compression techniques aiming to alleviate the bottleneck. quantization is a prevalent model compression technique that represents weights with lower precision. when the compression ratio is high, for instance, 4-bit quantization, the loss in accuracy becomes non-negligible. previous studies @xcite employ calibration sets to adjust the weights of llms for reduced quantization loss. nevertheless, the quantized llm may overfit the calibration set and exhibit poor performance on unseen data, as noted by @xcite . from another perspective, those works primarily focus on integer quantization, which overlooks the characteristic that model weights adhere to a gaussian distribution. @xcite observes the characteristic and introduces a non-uniform quantization schema to decrease quantization loss further. however, the conversion from non-uniform quantized values to full precision values relies on lookup tables. notably, accessing the lookup table can incur significant overhead during the memory-bound decoding stage. this work extends the ieee 754 floating-point standard to a low-bit schema. specifically, a floating point with an x-bit exponent and a y-bit mantissa is denoted as exmy. exmy differs from the previous standard by replacing infinity and nan with regular numbers. based on our observation, the conversion between exmy and fp16 can be accomplished by register-level operations, obviating the need for lookup tables. furthermore, we comprehensively analyze the quantization loss for various non-uniform quantization schemes, and fpe2m2 can reach the sweet spot of achieving lossless quantization of the minimum bit width. we comprehensively validate the qwen, llama, and deepseek distillation models framework on text, image, and audio tasks. we achieve a faster inference speed with negligible loss. our contributions are as follows: