Beyond Fixed-Length Calibration for Post-Training Compression of LLM s (2025)

Abstract: As large language models (LLMs) continue to grow in size, their practical deployment increasingly relies on a range of compression techniques, such as quantization, pruning, and low-rank approximation. Especially, post-training compression methods–which do not require re-training–have drawn considerable interest. Many recent methods leverage calibration data to capture magnitude or second-order characteristics of input activations. However, the role and significance of calibration data remain underexplored. In this study, we demonstrate that the sequence length of calibration data plays a crucial role in the effectiveness of post-training compression methods for LLMs. We then analyze input activations and find that, within the normalized hidden states, the embedding of the first token exhibits characteristics opposite to those of subsequent tokens. Building on this insight, we introduce state-aware length calibration, a technique that applies masking along the sequence axis, specifically targeting normalized hidden states. Experimental results show that our approach improves perplexity and zero-shot downstream tasks performance.

## introduction large language models (llms) have demonstrated remarkable capabilities across a wide range of natural language processing tasks, including text generation, translation, and question answering @xcite @xcite @xcite . nevertheless, their real-world deployment remains challenging due to their substantial computational and memory overhead @xcite . to overcome these limitations, researchers have developed various compression techniques designed to reduce model size and enhance inference efficiency, such as quantization, pruning, and low-rank approximation @xcite . notably, post-training compression methods have gained significant attention due to their ability to optimize model efficiency without requiring costly and time-consuming retraining. these techniques enable the direct compression of trained models, enhancing their practicality for deployment on resource-constrained devices, such as mobile devices and edge computing platforms. for instance, quantization methods, such as smoothquant @xcite and awq @xcite , reduce numerical precision to lower computational costs. pruning methods, such as sparsegpt @xcite and wanda @xcite , focus on identifying and removing redundant or less important weights from the model weights. additionally, low-rank approximation methods, such as asvd @xcite and svd-llm @xcite , decompose weight matrices into lower-dimensional forms. in the context of post-training compression framework, @xcite introduced a layer-wise optimization approach, where each layer is optimized independently to minimize performance degradation. specifically, the global compression task is decomposed into a series of layerwise subproblems, with each layer being optimized separately. this process leverages input activations derived from calibration data. in particular, calibration data is used to estimate input activation magnitudes or hessian (i.e., second-order) information, which is essential for achieving a more advanced and accurate compression process. despite its crucial role, calibration data remains an underexplored aspect of post-training compression. for example, calibration data is typically sourced from randomly selected web text or pretraining data, and constrained to a fixed length (e.g., 2048 tokens), under the observations that post-training compression methods are resilient to variations in calibration data distribution @xcite . however, @xcite conducted a systematic analysis of the influence of calibration data on compression performance. they revealed that the source and seed of the calibration data can largely influence on the results of both post-training quantization and pruning. additionally, @xcite stated that it is necessary to align the sequence length of the calibration data with that of the target task. in this paper, we present a systematic analysis of the impact of the sequence length of calibration data on post-training compression. our analysis involves four different llms, six post-training compression algorithms, and two calibration datasets to thoroughly evaluate how varying the sequence length influences the compressed model performance. our main contributions are as follows: ## sequence length of calibration data we investigate the influence of the sequence length of calibration data on the performance of the posttraining compression algorithms. tables 1 , 2 , and 3 present the perplexity results for post-training quantization, pruning, and low-rank approximation methods, respectively, according to the sequence length of calibration data. for post-training pruning, we evaluate the 4:8 structured pruning method, which is regarded as hardware-friendly. shorter calibration data provides unexpected benefits. when focusing on calibration sequences of 4 tokens and above, the experimental results reveal that very short sequences can yield low perplexity values, which directly correlate to improved performance. for example, the optimal sequence length has little correlation with the calibration dataset. tables 1 , 2 , and 3 indicate that the optimal sequence length is largely independent of the calibration dataset used, whether it is pile or wikitext. instead, it primarily depends on the methods and models employed. specifically, quantization generally exhibits superior perplexity when applied to shorter sequence lengths. in contrast, pruning tends to demonstrate optimal performance at mid-range sequence lengths. meanwhile, low-rank approximation often achieves the best results when dealing with longer sequences. however, it is important to note that these observed performance trends are not necessarily dictated by the inherent characteristics of the calibration dataset. instead, they are influenced by other factors, such as the model architecture and the specific compression methods. thus, these findings suggest that selecting an appropriate sequence length should be guided by the specific compression technique and llm rather than the specific calibration dataset. robustness hierarchy: quantization, pruning, and low-rank approximation. the experimental results clearly establish a hierarchy in terms of robustness when different compression methods are subjected to variations in calibration sequence length. specifically, the quantization method ( when sequence length is reduced to 1, the performance dramatically declines. although the discussion for the previous observations mainly focuses on sequences of 4 tokens or more, it is crucial to note the dramatic performance degradation that occurs when the calibration sequence is reduced to a single token. the results from ## analysis on input activation we conduct a detailed investigation of the input activations directly, with a particular emphasis on the first token (i.e., position 0). as depicted in figure 1 , a single layer typically contains four types of input activations: normalized hidden state in the attention module (attn) for w q/k/v , attention state for w o , normalized hidden state in the feed-forward network module (ffn) w gate/up , and intermediate state for w down . figure 2 illustrates the input activations of llama3-8b from position 0 to 63, with a period of 8, at four different layers: 0, 8, 17, and 31. specifically, (a) depicts the normalized hidden state feeding into the attention projections (w q , w k , and w v ), (b) depicts the attention state feeding into the attention output projection (w o ), (c) depicts the normalized hidden state feeding into the feedforward gate and up projections (w gate and w up ), and (d) depicts the intermediate state feeding into the feed-forward down projection (w down ). each sub-figure visualizes activation values (not their absolute values) across dimensions and token positions. we randomly select 64 samples from the pile dataset and average their activations. interestingly, position 0 consistently shows a distinct activation across most states and layers, even though no dedicated bos token is used in our setting. this implies that llms seem to have learned to process the first token in a distinct manner, no matter what it is. in fact, a similar phenomenon, referred to as "massive activations," has been observed @xcite . after passing through the initial few layers, a phenomenon occurs in which only specific dimensions of the hidden state have extremely large magnitudes, and this is observed at the first position. this can also be inferred from figures 2(a ) and (c). although we have provided the normalized hidden state, the dimensions with large magnitudes are being propagated through the residual connection. moreover, @xcite demonstrated that when massive activations are set to zero, llms completely fail to function. based on this finding, we believe that the statistics of the first token should be adequately considered during calibration. in the leftmost of figures 2(a ) and (b), it can be observed that the activations of position 0 behave similarly to those of other positions. to quantify this, we utilize the jaccard similarity, which is a metric for measuring the similarity between two sets by dividing the size of their intersection by the size of their union. we first calculate the average of the activations for all positions except the first token. next, we extract the indices of the top 30 maximum values and the top 30 minimum values from both the position 0 activations and the averaged activations. figure 3 describes the jaccard similarity between the extracted indices, where max(0) and min(0) refer to the indices of the top 30 maximum values and the top 30 minimum values from position 0, respectively. similarly, max(others) and min(others) are defined in the same manner. a high jaccard similarity between max(0) and max(others) (i.e., blue line) or between min(0) and min(others) (i.e., sky-blue line) means that the first token and the subsequent tokens share similar activation patterns. on the other hand, a high jaccard similarity between max(0) and min(others) (i.e., red line) or between min(0) and max(others) (i.e., orange line) means that the first token and the subsequent tokens exhibit opposite activation patterns. ## state-aware length calibration building on prior observations and evidence indicating that the first token plays a pivotal role in the performance of llms @xcite @xcite , we propose a simple calibration technique termed state-aware length calibration. specifically, when the sequence length of calibration data is l, only the initial lr tokens of the normalized hidden state in both modules are employed for calibration, where r denotes a predetermined ratio. in contrast, for other states, the full sequence length l is retained, as illustrated in figure 1 . by concentrating calibration efforts on the initial critical tokens, our method aims to capture the most salient features, thereby improving the effectiveness of the calibration process. tables 4 , 5 , and 6 present the perplexity results obtained from applying state-aware length calibration to post-training quantization, pruning, and lowrank approximation methods. our experimental findings indicate that leveraging partial sequence lengths (i.e., @xmath1. although there exist certain cases where the observed performance gains are relatively modest, we would like to emphasize the significance of state-aware length calibration as a promising and effective strategy. this is because our proposed approach incurs no additional computational cost, making it a practical and efficient approach for improving the performance of compressed models. ## conclusion in this study, we investigate the critical role of calibration data sequence length in improving the effectiveness of post-training compression methods for llms. our analysis reveals that the normalized hidden states of the first token behave oppositely to those of subsequent tokens. motivated by this observation, we introduce state-aware length calibration, a novel technique that uses a subset of initial tokens for calibration. experimental results demonstrate that this approach generally enhances perplexity across various post-training compression methods and llms. notably, as state-aware length calibration incurs no additional computational cost, we consider it a practical and simple strategy for optimizing post-training compressed llms.