Pruning Weights but Not Truth: Safeguarding Truthfulness While Pruning LLM s (2025)

Abstract: Neural network pruning has emerged as a promising approach for deploying LLMs in low-resource scenarios while preserving downstream task performance. However, for the first time, we reveal that such pruning disrupts LLMs’ internal activation features crucial for lie detection, where probing classifiers (typically small logistic regression models) trained on these features assess the truthfulness of LLM-generated statements. This discovery raises a crucial open question: how can we prune LLMs without sacrificing these critical lie detection capabilities? Our investigation further reveals that naively adjusting layer-wise pruning sparsity based on importance inadvertently removes crucial weights, failing to improve lie detection performance despite its reliance on the most crucial LLM layer. To address this issue, we propose Truthful Pruning aligned by Layer-wise Outliers (TPLO), which places greater emphasis on layers with more activation outliers and stronger discriminative features simultaneously. This preserves LLMs’ original performance while retaining critical features of inner states needed for robust lie detection. Moreover, we introduce a prompting rule to enrich the TruthfulQA benchmark for better calibrating LLM pruning. Empirical results show that our approach improves the hallucination detection for pruned LLMs (achieving 88% accuracy at 50% sparsity) and enhances their performance on TruthfulQA.

## introduction large language models (llms) @xcite are remarkably impressive across a wide range of 1 in this paper, we adopt the definition of "hallucinations" from recent works @xcite , referring to instances where llms produce fluent, instructioncompliant, yet untruthful responses. consequently, we use the terms "lie detection" and "hallucination detection" interchangeably throughout this work. figure 1 : each curve represents the layer-wise ratio of between-class variance to within-class variance for activations corresponding to true and false statements. this ratio is averaged across all dimensions within each llm layer, indicating that layers with a higher ratio contain more discriminative features for distinguishing between true and false statements, whereas layers with a lower ratio have fewer. we define this metric as layerwise separability of true and false distribution (lsd). three key takeaways: i) original models (unpruned llama3.1-8b-instruct) have the best ability to separate true/false statements. ii) moderate pruning (e.g., less than 0.5 sparsity) retains reasonable performance, but heavy pruning (e.g., 0.65 sparsity) significantly degrades separation ability. iii) the most useful layers for classifying true/false statements seem to be consistently around layer 10-15 no matter what sparsity is. we study this question by training classifiers on the internal activations of pruned llms to judge whether a given statement is true or false, using both supervised @xcite and unsupervised techniques @xcite . we discover that after being unstructuredly pruned via wanda @xcite ) at sparsity of 50% followed with @xcite , the quality of llms' internal activations will deteriorate, leading to a less robust lie detector shown in figure 2 . we conjecture that applying a uniform pruning ratio across all layers, where each layer is pruned at the same sparsity, is detrimental to the training of robust lie detectors, as intermediate activation features might contribute differently to lie detection across layers. to validate this, we visualize the lsd of the original llama3.1-8b-instruct in figure 1 and observe that each llm layer exhibits varying degrees of discriminative quality in its internal states, suggesting that some layers are more effective at distinguishing between true and false statements while others are not. based on this observation, we construct a baseline method, separability weighted layer-wise sparsity (swl), which adjusts each layer's pruning sparsity inversely proportional to its layer-wise separability. however, simply applying swl may inadvertently prune more important weights, as suggested by owl @xcite which highlights that llm outlier distributions across layers follow a distinctly non-uniform pattern that does not fully align with lsd. recognizing owl as a valuable indicator for effectively optimizing layer-wise sparsity strategies in llm pruning, we propose truthful pruning aligned by layer-wise outliers (tplo). tplo enhances layer-wise pruning by aligning lsd with outlier ratio distributions, ensuring a more effective sparsity allocation. our approach is based on the insight that greater emphasis should be synchronously placed on layers with a higher prevalence of outliers and more discriminative activation vectors, ensuring that the original performance of llms is maintained while preserving more internal features essential for training lie detectors. furthermore, we propose a novel prompting rule to enrich the truthfulqa benchmark @xcite as extra calibration data to help prune llms inspired from @xcite that c4, one of data sources to pretrain llms, is not the optimal choice for calibrating llm pruning. in summary, the contributions of this paper are as follows: ## mitigation strategy the analysis in subsection 3.2 highlights that swl induces a layer-wise sparsity distribution that significantly differs from owl's, resulting in a performance decline compared to wanda, as shown in figure 2 . our approach builds on the insight that modifying swl based on owl can guide enhancement of the pruning effect, as owl helps identify llm layers with more significant activation scores. additionally, swl remains essential for preserving the internal states of mid-layers, ensuring that more inner features are maintained for lie detection tasks. thus, we propose a novel framework tplo, i.e., truthful pruning aligned by layerwise outliers, for pruning llms such that they maintain discriminative internal features to train robust lie detectors, which can be seamlessly integrated with existing hallucination detection methods @xcite . additionally, inspired by @xcite that c4, one of the pretraining data sources for llms, might not the optimal calibration set for pruning llms and pruning with downstream data could help improve performance, we utilize gpt-4o @xcite to enrich the truthfulqa benchmark @xcite to become the supplementary calibration data for pruning. ## conclusion in this work, we propose tplo, a novel pruning method that places greater emphasis on layers with more activation outliers and stronger discriminative features simultaneously. our approach preserves llms' original performance while maintaining essential inner states needed for robust lie detection. moreover, we introduce a prompting rule to enrich the truthfulqa benchmark for better calibrating llm pruning. comprehensive experiments demonstrate that our approach improves the hallucination detection for pruned llms and enhances their performance on the truthfulqa benchmark. our findings underscore the importance of integrating truthfulness assessments into the development of pruning llms to ensure their reliability across real-world applications.