Improving Efficiency in Large Language Models via Extendable Block Floating Point Representation (2025)

Abstract: Large language models (LLMs) have revolutionized natural language processing (NLP) tasks, yet their increasing size poses substantial challenges in terms of computational and memory resources. Block floating-point (BFP) arithmetic offers an effective solution by leveraging the strengths of both floating-point and fixed-point representations, leading to reductions in both storage and computational overhead. However, current low-bit BFP quantization approaches often struggle to handle extreme outliers, leading to significant accuracy degradation. To overcome this limitation, we introduce Extendable Exponent Sharing (EES), a novel BFP representation that extends the exponent bit width to capture a wider dynamic range. EES achieves this by embedding extendable exponent bits into the least significant mantissa bits, thereby increasing the shared exponent’s bit width without incurring additional storage costs. To optimize the trade-off between accuracy and energy efficiency, EES employs a design space exploration strategy to optimize the configuration of extendable exponent bit widths. Experimental results show that EES outperforms representative baselines in both accuracy and computational efficiency.

## introduction large language models (llms) have achieved remarkable success across a variety of natural language processing tasks, such as intelligent assistants, machine translation, and sentiment analysis @xcite @xcite . however, as the size of models continues to scale, the computational and memory requirements grow substantially @xcite . block floating point (bfp) representation provides an effective solution to address these challenges, balancing precision with efficiency, making it particularly well-suited to meet the resource-intensive * corresponding author: bosheng liu and jigang wu demands of llms @xcite . by grouping multiple values under a shared exponent, bfp enables efficient storage and calculation of large model parameters with minimal overhead @xcite . in comparison to both fixed-point and floating-point formats, bfp offers distinct advantages, especially in large-scale computations. it minimizes memory usage by sharing an exponent across values, making it more storage-efficient than full-precision floating point, while providing greater numerical stability than fixed-point representation @xcite . recent advancements in low-bit bfp quantization have shown great promise in reducing memory and computational costs for llm applications @xcite . bfp effectively balances numerical precision and resource efficiency @xcite , but its limited exponent range in low-bit formats presents significant challenges. for example, in an 8-bit bfp representation using the e3m4 format, a block of numbers shares a 3-bit exponent and uses 4-bit mantissas. if the shared exponent is set to 4 (2 4 = 16), the largest representable value in the block is 240 (15 × 16 = 240). any value exceeding this, such as 300, results in overflow, as the shared exponent cannot accommodate larger values within the given bit width. this limitation becomes particularly problematic for extreme outliers, which are prevalent in the parameters and activations of llms, as highlighted in fig. 1 . the inability to accurately handle these outliers often causes significant accuracy degradation @xcite @xcite . solving this issue is essential for enabling the broader application of bfp in high-precision llm workloads. to mitigate the challenges posed by extreme outliers in low-bit bfp quantization, various techniques have been proposed. superweight @xcite tackles the issue by identifying and preserving critical outlier weights during quantization, achieving a balance between compression efficiency and accuracy. similarly, smoothquant @xcite adopts a per-channel scaling approach to redistribute value ranges, effectively shifting the quantization burden of outliers from activations to weights while preserving mathematical equivalence. building on these strategies, bie @xcite introduces a dual-exponent mechanism to address outliers, ensuring improved accuracy. however, this method incurs additional resource overhead due to the inclusion of flag bits during computational implementation. despite these advancements, existing methods only partially alleviate the challenges of low-bit bfp quantization @xcite . the inherent limitation of a narrow exponent width remains a critical bottleneck, restricting the efficient handling of extreme values and limiting computational performance. in this work, we introduce extendable exponent sharing (ees), a post-training quantization approach based on bfp designed to improve dynamic range and computational efficiency. ees extends the traditional shared exponent method by increasing the exponent bit width to cover a broader range of values. however, directly expanding the exponent width increases the storage requirements. to address this, we observe that changes in lowerorder bits have a minimal effect on computations and model accuracy. for example, two 8-bit bfp numbers-1101.1010 and 1101.1001-differing only in the least significant bit show a minimal impact on precision, indicating that small changes in lower-order bits do not significantly affect accuracy. using this insight, ees embeds the extended exponent bits into the least significant bits of the partial mantissas. these bits are then combined with the shared exponent to form an extended exponent, as shown in fig. 2 . since this approach affects the precision of mantissas, we analyze the quantization error introduced by ees compared to standard bfp. to achieve an optimal balance between accuracy and computational efficiency, we use a design space exploration strategy to obtain the best configurations for both activations and weights. evaluations in llms show that ees significantly improves both accuracy and energy efficiency. key contributions of this work include: ## related work quantization is a key technique used to compress llms by lower-precision data formats, significantly reducing memory usage and computational costs @xcite . the two dominant data formats in this space are fixed-point @xcite and bfp @xcite representations. fixed-point representation simplifies operations like multiplication and accumulation compared to floating-point formats, but it has limited precision due to the fixed offset and scaling. @xcite differently, bfp combines the computational efficiency of fixed-point arithmetic with the dynamic range of floating-point formats, offering an effective balance between precision and performance @xcite . recent progress in bfp quantization, including techniques like dbps @xcite , static bfp @xcite , and msfp12 @xcite , has shown its ability to achieve high accuracy, efficient computation, and reduced storage costs. additionally, research into low-bit bfp representations aims to improve these advantages. for example, fast @xcite introduces low-bit bfp quantization for activations and weights, supporting efficient matrix multiplications with reduced precision. similarly, bucket @xcite and figna @xcite adopt 11bit and 8-bit bfp formats, respectively, to enhance computational performance and energy efficiency. in the bfp representation, data can be classified into two categories: "normal values," which are accurately represented, and "extreme outlier values," which fall outside representable range and require approximation through shifting or scaling. these outliers are a major challenge in low-bit bfp quantization, as they can significantly degrade model accuracy. techniques like awq @xcite and superweight @xcite identify and preserve "super weights" -critical parameters with extreme values -to mitigate the negative effects of outliers on model quality. @xcite introduces dual shared exponents to manage both normal values and outliers in both activations and weights, ensuring high accuracy for llm models. while these methods address outlier challenges, the limited bit width of the shared exponent in bfp remains a key limitation for representing a wider range of values. we propose a scalable shared exponent strategy that extends the range of numerical representation and improves the performance of low-bit bfp quantization for llms. ## preliminary as shown in fig. 2(a) , a set of n floating-point numbers, denoted as x, can be given by: when converting floating-point data into bfp representation, we obtain the new data set x ′ , as illustrated in fig. 2(b) , which is given by: where e ′ s is the shared exponent, calculated as: for each mantissa in this bfp representation, m ′ i is computed by shifting the original mantissa m i based on the difference between the maximum shared exponent (e ′ s ) and the individual exponent (e i ), using the right shift operation: assuming the value of input activations and weights are known and the rounding-to-nearest method is used @xcite , the mean of α is zero, and the error variance (σ 2 α ) can be formulated as ## conclusion in this paper, we propose an extendable exponent sharing (ees) method to overcome the limitations of existing low-bit quantization techniques. ees enhances dynamic range capture by embedding extendable exponent bits into lower-order bits of mantissa, without introducing additional overhead. to further optimize performance, we design a spatial exploration strategy to determine optimal bitwidth configuration for both extendable exponent and mantissa. comprehensive evaluation show that ees outperforms representative baseline methods in terms of both accuracy and energy efficiency.