Divergent Token Metrics: Measuring degradation to prune away LLM components – and optimize quantization (2024)

Abstract: Large Language Models (LLMs) have reshaped natural language processing with their impressive capabilities. However, their ever-increasing size has raised concerns about their effective deployment and the need for LLM compression. This study introduces the Divergent Token Metrics (DTMs), a novel approach to assessing compressed LLMs, addressing the limitations of traditional perplexity or accuracy measures that fail to accurately reflect text generation quality. DTMs measure token divergences that allow deeper insights into the subtleties of model compression, in particular, when evaluating components’ impacts individually. Utilizing the First Divergent Token Metric (FDTM) in model sparsification reveals that 25% of all attention components can be pruned beyond 90% on the Llama-2 model family, still keeping SOTA performance. For quantization, FDTM suggests that more than 80% of parameters can be naively transformed to int8 without special outlier management. These evaluations indicate the necessity of choosing appropriate compressions for parameters individually—and that FDTM can identify those—while standard metrics result in deteriorated outcomes.

## introduction cutting-edge large language models (llms) based on the transformer architecture @xcite have revolutionized natural language processing with their exceptional performance, notably exemplified by the gpt series @xcite https://github.com/aleph-alpha/divergent_tokens @xcite @xcite ) in text generation. however, these models have grown massively, even exceeding half a trillion parameters @xcite . although the large number of parameters aid in early training convergence, their practical utility and true necessity remain unclear. in particular, for the attention mechanism, it was hinted that after some training convergence, certain heads dominate the inference process @xcite . compression strategies such as sparsification and quantization can enhance the efficiency of the parameters. current metrics, however, either average too coarsely, such as perplexity, or are by design too specific, such as standard nlp benchmarks. both fail to capture the diverging performance nuances introduced early on by the compression as they ignore the actual discontinuous text generation process. this, however, is the main application of the final model, and so we argue that they are therefore insufficient measures for the performance of the compressed model. this misalignment can lead to unwanted subtle discrepancies in generation, such as grammatical errors or a mismatch in numbers, that cause subsequent divergences. as we will show, this even occurs when overall metrics, such as perplexity, appear satisfactory (cf. prop. 3.2, sec. 4, tab. 1). to meet these challenges, we introduce the family of divergent token metrics (dtms) in sec. 3. these metrics are tailored to measure the model divergence of llms throughout the compression process and in relation to the actual generation procedure on a token basis, as shown in fig. 1 . we figure 1 : illustration of a diverging generation process. given the 3-token prefix as prompt, a baseline and its compressed model generate 8 subsequent tokens. our proposed metric points to the first divergent token (fdt). the fdt may cause further divergence during the iterative generation process. note how both models score the same perplexity value, as it does not reflect the actual sampling process (cf. fig. 2 , sec. 4 for an empirical exploration). specifically, our proposed individual component sparsification indicates significant differences in component utilization across layers. for the first time, we show that 25% percent of the models' attention components can be pruned beyond 90%, and several even entirely removed, while preserving a single-digit perplexity. consequently, a sparse matrix format can be employed to accelerate computational efficiency. likewise, for precision reduction, we show that sorting components by fdtm coincidentally correlates to sorting by their induced number of outliers when being naively converted to int8. fdtm identifies the optimal 80% of the components that maintain overall performance without specific outlier handling. the observed decline in performance with more outliers, and the significant influence of specific components on those, suggests reevaluating the applied normalization methods throughout the model. we demonstrate that this level of precision goes beyond what standard perplexity and conventional nlp benchmarks can achieve. the proposed divergent token metrics closely reflect the generation process and so can be a measure to foster confidence in the deployed compressed models. ## compression principles model compression aims to reduce the hardware resources needed to operate the model. however, doing so may sacrifice the accuracy of the model. to keep the loss as small as possible, a corrective measure is typically used. here, we discuss the most commonly used concepts and state-of-the-art methods for llm sparsification and quantization. outlier and hessians. most model compression methods rely either on the separation of outliers @xcite or the computation of a hessian matrix @xcite . outliers usually refer to significantly larger values in magnitude occurring either in the weight matrix directly or in the activations during a forward pass. as most computations are linear matrix multiplications, such outliers strongly influence the remaining entropy contained in consecutive computations. in the case of sparsification, outliers should be left intact, and values with the least magnitude, which are consequently the least influential, should instead be masked @xcite . on the other hand, hessian matrices can be applied to correct errors @xcite . they can effectively be approximated by computing backpropagation gradients for a small number of samples and represent a second-order approximation to reconstruct the original model. sparsification. the goal of sparsification is a reduction in the total number of weights and therefore a distillation of the relevant computation. typically, this method is divided into "structured" and "unstructured" pruning. structured-pruning aims to locate dynamics, such as the irrelevance of an entire layer or dimension for a given use case, and prunes these entirely. unstructured-pruning usually refers to weight masking, that is, setting irrelevant weights to 0. high levels of sparse matrix computations could result in more efficient kernels and computations. in scenarios where masks exceed a 90% threshold, the implementation of a specialized sparse matrix format becomes feasible. this format predominantly stores the indices of non-zero weights. although some additional storage is required for these indices, the overall requirement is reduced due to the exclusion of zero values. moreover, this approach substantially improves computational performance. magnitude pruning selects the masking of weights based only on their magnitudes. this is fast to compute but significantly degrades model performance when pruning large amounts simultaneously. to resolve this issue, wanda @xcite proposes to sample a small amount of data and incorporates activations of the forward pass. this was shown to generate more effective one-shot pruning masks. sparsegpt @xcite computes iterative hessian approximations to select the lowest impact weights and correct the remaining. note that the incorporation of activations can, to some extent, be interpreted as a form of training. moreover, despite these efforts, one-shot pruning has not yet produced directly usable models without further final fine-tunings. this is in particular the case for the high sparsity levels beyond 70% that we target. finally, there has not yet been any investigation of the individual components. quantization. model quantization refers to the reduction of the precision of the used numerical format. usually, llms are trained in 16-bit floating point (fp16) and converted to 8-bit integer (int8) representations. the naive conversion of float matrices to integers is absmax rounding. this divides a number by the maximum value occurring in the matrix and multiplies by the largest available integer, as such, it spans a uniform representation grid. the largest float value is stored and multiplied for dequantization. the most prominent methods for minimizing the introduced rounding errors are llm.int8 and gptq. dettmers et al. (2022) introduced llm.int8(), which identifies vectors containing outliers and retains them in their original fp16 form during the matrix multiplication of a forward pass. the vectors lacking outliers are fully quantized. the int8 weights and activations during the forward pass are subsequently multiplied and dequantized afterward. this allows them to be integrated with the fp16 representation of the outliers. through empirical investigation optimizing the trade-off between degradation in perplexity and the number of outliers preserved in fp16, they fixed an absolute outlier threshold. the gptq framework offers a more robust quantization approach, in particular, to different integer bit precisions. it does not rely on any outlier detection mechanism or mixed precision computations-matrix multiplications with the weights are fully performed using integers. @xcite introduce an efficient hessian approximation and iteratively quantize the weights of the matrices while performing error corrections on the remaining weights. ## model divergence metrics perplexity fails to identify minor variations in model degradation at an early stage. this behavior is depicted in fig. 1 and 2 and discussed in sec. 3.5 and 4 in more detail. to assess model divergence and enhance the model compression process, we introduce token-based metrics specifically designed to detect those nuances occurring in early compression stages. we start by establishing our notation and presenting the perplexity metric (ppl). subsequently, we introduce an enhanced variant of ppl and propose the share of divergent tokens metric (sdtm) and first divergent token metric (fdtm). we conclude by discussing the advantages of each metric compared to traditional measures based on perplexity when assessing the degradation of the generative performance of compressed models. ## token metrics improve model compression we will demonstrate in the following how the proposed metrics provide novel insights into the efficiency of the architecture of llms and establish benchmarks for model compression. throughout all experiments, we outperform standard ppl as a ranking metric. more precisely, we apply componentwise probing to sparsification to determine individual sparsity rates. interestingly, the model tends to completely remove components of the attention mechanism in certain layers. in total, 40 out of 160 attention components are sparsed beyond 90% and 15 removed completely. for quantization, on the other hand, we show how component selection significantly influences the overall number of model outliers. for the first time, almost 10% of the model components can be converted to 4-bit integer representations without significant model degradation. ## conclusion we introduced the divergent token metrics (dtms), a tailored approach to evaluate the performance differences of compressed generative models. in particular, dtms respect the usually applied greedy sampling procedure to generate predictions. we proved that dtms achieve appropriate metric bounds and are not affected from catastrophic artefacts that perplexity-based metrics encounter. we constructed algorithms for iterative sparsification and quantization processes based on dtm probing of individual components. using dtms, we achieved an outperforming 75% sparse version of llama2-13b and successfully converted 80% of the llama2-7b components naively to int8.