AMXFP 4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM Inference (2025)

Abstract: As large language models (LLMs) grow in parameter size and context length, computation precision has been reduced from 16-bit to 4-bit to improve inference efficiency. However, this reduction causes accuracy degradation due to activation outliers. Rotation-based INT4 methods address this via matrix calibration, but they introduce multi-hour overheads and leave key computations in full precision. Microscaling (MX) floating-point (FP) formats offer fine-grained representation with a shared scale, enabling fully quantized matrix multiplications through direct casting without calibration. However, existing research shows unsatisfactory empirical results for MXFP4 inference, and the robustness of MX formats remains largely unexplored. In this work, we uncover the fundamental tradeoffs of the MX format: while it effectively suppresses activation outliers, it does so at the cost of increased group-wise asymmetry. To address this, we propose AMXFP4, a 4-bit asymmetric FP format that handles both issues using asymmetric shared scales, without requiring calibration. Our custom MAC engine adds negligible hardware cost while improving accuracy: AMXFP4 outperforms MXFP4 by 3% on VQA and exceeds rotation-based methods by 1.6% on CSQA. It also surpasses recently deployed commercial MXFP4 variants. Code: https://github.com/aiha-lab/MX-QLLM

## introduction multi-modal large language models (llms) are widely used in advanced natural language processing tasks, including chatbots, long-document question-answering, and visual graph interpretation @xcite . to enhance their capabilities, llms have been significantly scaled in both parameter size and context length @xcite . for example, llama3 (ai@meta, 2024) now features 405 billion parameters and supports context lengths of up to 128k tokens. as shown in fig. 1(a) , this scaling results in peta-flop-level computational demands during the prefill phase, where the model processes user context before inference. leading computing platforms have focused on bit-precision scaling to meet the computational demands of llms @xcite @xcite . reducing operand bitwidths improves area and energy efficiency in arithmetic operations @xcite , enabling higher computation density in accelerators. as shown in fig. 1(b ), nvidia's tensor cores double computation speed by lowering multiply-accumulate (mac) precision from fp16 to fp8 @xcite and from int8 to int4 @xcite . recent research explores activation and weight quantization to improve llm inference efficiency by leveraging hardware precision scaling. however, quantizing both weights and activations to int4 often degrades accuracy due to activation outliers @xcite . rotation-based transformations mitigate this by making activations more quantizationfriendly @xcite , with approaches like quarot @xcite significantly reducing llm perplexity in int4 inference (fig. 1(c )). despite these benefits, rotation-based methods require extensive calibration, leading to overfitting risks @xcite (cf. an alternative approach to quantization introduces reduced-precision formats that enable calibration-free data-type conversion (i.e., direct casting). for instance, the latest nvidia tensor core @xcite supports the microscaling (mx) format, introduced by the open compute project (ocp) @xcite , which groups low-precision elements under a shared scale to mitigate dynamic range limitations (fig. 1 (b), (d)). as shown in fig. 1(c ), (e), mxfp4 achieves full matrix quantization with minimal perplexity degradation compared to int4, without requiring data rotation. this is due to its fine-grained quantization, which enhances value representation precision. however, mxfp4 still lags behind the 16-bit baseline in perplexity and performs worse when combined with data rotation, and the root causes of this destructive interaction are mainly unexplored. this work uncovers a key trade-off in the mx format: while it effectively suppresses activation outliers, it increases group-wise asymmetry. grouping activation tensors into small micro-scaled units mitigates outliers, similar to rotation methods, but enables direct-cast inference. however, this grouping amplifies data asymmetry, necessitating an asymmetric numerical representation. to address this, we propose amxfp4, a microscaling floating-point format designed for robust 4bit llm inference, which effectively handles activation outliers through micro-scaled asymmetric data representation. by employing an fp8 shared scale for both weights and activations, amxfp4 achieves quantization error rates close to ideal lloyd-max quantization. to validate its broad applicability, we evaluate amxfp4 across multi-turn conversation, long-context inference, and visual question-answering (vqa) tasks on decoder-only llms, vision-language models, and an encoder-decoder model. results show that amxfp4 enables calibration-free, direct-cast 4-bit inference, outperforming mxfp4 and leading rotation-based quantization methods. additionally, amxfp4 performs better than the recently deployed commercial mxfp4 format (nvfp4) @xcite . our contributions can be summarized as follows: ## microscaling for taming outliers we systematically analyze activation outliers across various llms using representative statistical measures-kurtosis and mean-to understand the effects of microscaling (i.e., reducing a quantization group to 32 elements). kurtosis, the fourth standardized moment, is commonly used to assess the prevalence of outliers @xcite , while the mean reflects asymmetry within each group. we use box plots of kurtosis and mean to examine the value distribution within groups, which are subject to quantization using a shared scale. -0.5 0.5 0 mean 0 kurtosis group: row 2048 512 128 32 -1 ■ opt-125m ■ opt-1.3b ■ opt-2.7b ■ opt-6.7b (a) kurtosis of ffn1 activations kurtosis ■ llama2-7b ■ llama3-8b ■ qwen-7b ■ mistral-7b (b) kurtosis of ffn2 activations (c) mean of ffn1 activations (d) mean of ffn2 activations row 2048 512 128 32 mean 4000 0 10000 0 -2 1 2 0 -1 kurtosis mean -2 1 2 0 (e) kurtosis after data rotation (f) mean after data rotation ■ llama2-7b qkv-proj input ■ after data rotation 3000 figure 2: data characteristics based on (a-d) types of llm, layer and (e-f) data rotation across group sizes. ## asymmetric microscaling format the findings from sec. 3 motivate the development of a new microscaling format that inherently supports asymmetric data representation. in this section, we explore the design space of the microscaling data format (p i and s) alongside considerations for asymmetric quantization schemes. ## experiments in this section, we compare amxfp4 with other formats and rotation-based methods. unless otherwise specified, all experiments use the proposed fp8 shared scale across all formats (including int4, mxfp4, and amxfp4) for a fair comparison and quantize input operands for all decoderlayer matrix multiplications. further details on quantization settings and benchmark descriptions are provided in appendix c. ## conclusion to meet the computational demands of large language models (llms) with extended contexts, we introduce asymmetric microscaling 4-bit floating-point (amxfp4), which uses asymmetric shared scales to handle outliers and quantization asymmetry. amxfp4 provides direct 4-bit inference with high accuracy, outperforming mxfp4 and other techniques for efficient, calibration-free inference. acknowledgement this work was supported by institute of information & communications technology planning & evaluation (iitp) (under the artificial intelligence semiconductor support program to nurture the best talents, iitp-2025-rs-2023-00253914, and no.rs-2025-02214497, development of low-level optimization program api technology for ai semiconductors) and national research foundation of korea (nrf) (no. rs-2025-00561961) grant funded by the korea government (msit).