Revisiting Pruning vs Quantization for Small Language Models (2025)

Abstract: Deploying language models on resource-constrained devices, such as mobile phones, wearables, and on-device AI assistants, demands compact, efficient models without sacrificing performance. Compressing Small Language Models (SLMs) is particularly suited for these scenarios, yet their compression dynamics remain underexplored compared to Large Language Models (LLMs). We systematically evaluate leading post-training pruning (SparseGPT, Wanda) and quantization (GPTQ, AWQ) methods across six SLMs from 0.5 to 3.8B, seven languages, and seven downstream tasks. Our results show that quantization consistently outperforms pruning in preserving model fidelity, multilingual perplexity, and reasoning accuracy. However, quantization’s advantages diminish on complex knowledge and reasoning tasks like OpenBookQA, highlighting a disconnect between compression fidelity and downstream task performance. Notably, trends observed in LLMs (e.g., Wanda’s competitive performance to SparseGPT) do not generalize to SLMs. For practitioners, we recommend prioritizing quantization (particularly AWQ) for SLM compression and caution against relying on a single metric.

## introduction the pervasive demand for intelligent systems in diverse applications, from resource-constrained devices to privacy-sensitive offline deployments, necessitates compact models and efficient inference @xcite @xcite . small language models (slms) offer distinct advantages through their efficiency, faster inference, and lower memory footprint, making them well-suited for deployment in constrained environments @xcite . despite their smaller parameter count compared to large language models (llms), further compression is crucial and beneficial to accommodate figure 1 : layer-wise activation snr for quantized and pruned models. higher snr indicates better compression fidelity. both pruning (left column) and quantization (right column) exhibit noticeably reduced snr at similar layers, but quantization consistently achieves higher snr than pruning across models. compression techniques such as pruning and quantization have been extensively explored for llms; however, the effectiveness of these methods when directly applied to slms remains unclear. particularly, advanced pruning methods like sparsegpt @xcite and wanda @xcite , and state-of-the-art quantization methods like gptq @xcite and awq @xcite , have not been thoroughly compared in the context of slms @xcite @xcite . 1 this gap motivates a principled study to uncover generalizable patterns, which is currently hindered by inconsistent benchmark usage and the lack of comprehensive evaluations in existing work. we conduct extensive experiments across multiple slms and compression ratios, evaluating three dimensions: (1) compression fidelity to assess the activation changes after compression, fig 1 as an example of layer-wise snr comparison, (2) perplexity to evaluate language modeling capability for multiple languages, and (3) downstream tasks to investigate the reasoning capability. the results indicate that quantization consistently outperforms pruning across all three evaluation dimensions, with particularly strong advantages under high compression. it offers better preservation of compression fidelity and language modeling performance, while its superiority on downstream reasoning tasks is less consistent and varies across models and task types. ## methodology we conduct a comprehensive evaluation of two state-of-the-art pruning and quantization methods each, under two commonly used compression settings (sec. 2.1). we experiment with five popular slms (sec. 2.3), spanning three-dimensional evaluations: two compression fidelity metrics, perplexity across seven languages, and accuracy on seven downstream tasks (sec. 2.2). with the fullsize baseline, this comprehensive setup results in a total of 710 evaluations across methods, models, and languages. ## discussion our comprehensive comparison suggests the superior choice of quantization over pruning for slms. although the selected pruning and quantization techniques follow a similar local activation reconstruction error minimization to retain performance @xcite , we empirically observe that by removing entire weights, pruning impacts performance more severely than quantization. quantization introduces small, uniform perturbations (noise) to weights, preserving overall model structure and learned representations @xcite . pruning, however, removes entire connections, significantly altering neuron connectivity and functionality @xcite @xcite ). while quantization reduces numerical precision, it keeps neuron connectivity intact, maintaining higher activation snr than pruning @xcite , theoretically explaining its empirical robustness @xcite .