ALPS : Attention Localization and Pruning Strategy for Efficient Adaptation of Large Language Models (2025)

Abstract: Aligning general-purpose large language models (LLMs) to downstream tasks often incurs significant training adjustment costs. Prior research has explored various avenues to enhance alignment efficiency, primarily through minimal-data training or data-driven activations to identify key attention heads. However, these approaches inherently introduce data dependency, which hinders generalization and reusability. To address this issue and enhance model alignment efficiency, we propose the Attention Localization and Pruning Strategy ALPS, an efficient algorithm that localizes the most task-sensitive attention heads and prunes by restricting attention training updates to these heads, thereby reducing alignment costs. Experimental results demonstrate that our method activates only 10% of attention parameters during fine-tuning while achieving a 2% performance improvement over baselines on three tasks. Moreover, the identified task-specific heads are transferable across datasets and mitigate knowledge forgetting. Our work and findings provide a novel perspective on efficient LLM alignment.

## introduction pre-trained large language models (llms) have demonstrated impressive performance across various tasks @xcite @xcite . a way to harness the potential of pre-trained foundation models on downstream tasks is task alignment, which mainly incorporates tailored task knowledge. recently, task alignment has garnered considerable attention in both industry and academia with a diverse range of specialized models having been developed, spanning domains such as mathematics @xcite , code generation @xcite , medical diagnostics @xcite and protein structure prediction @xcite . however, despite the effectiveness of task alignment, it still poses a significant resource-intensive challenge. the alignment process demands extensive efforts to construct task-specific instruction datasets and training adjustments, and further, its computational demands exacerbate the costs @xcite . these challenges highlight the pressing need for more efficient alignment strategies that not only minimize resource consumption costs but also maintain or even enhance model performance @xcite . to address the efficiency challenges of model alignment, recent llm studies are interested in attention mechanisms and attempt to localize taskspecific attention heads @xcite @xcite @xcite . however, these methods mainly rely on task-specific data to activate model parameters for attention head localization, inadvertently introducing data dependency to identified heads, which not only hinders generalization but also complicates reusability. moreover, these approaches overlook the intrinsic functionality of the model weight parameters, which inherently encode task-relevant information @xcite , and can be leveraged to identify task-sensitive attention heads without relying on activation data. in this work, to enhance alignment efficiency, utilize weight parameters, and introduce reusability of identified attention heads, we propose the attention localization and pruning strategy (alps), a heuristic-guided algorithm that localizes task-sensitive attention heads and prunes by restricting attention training updates to these heads only. this algorithm draws inspiration from prior work leveraging model weight parameters to localize task-sensitive attention heads @xcite @xcite , as shown in fig 1 . unlike previous work utilizing whole model parameters, alps focuses on the distribution of each attention head. specifically, given a base model and its task-related model, we extract and normalize the distribution of each head, then quantify the distributional shift between corresponding heads using our proposed parameter alignment distribution score. the heads with the highest scores are then selected as task-sensitive heads, while the remaining heads are frozen during fine-tuning. this yields a pruned, task-specialized model with enhanced efficiency. empirical results demonstrate that our method updates only 10% of attention parameters during fine-tuning while achieving a 2% performance gain over baselines across general, math, and code tasks, as shown in figure 2 . further analysis reveals that the selected task-sensitive heads exhibit transferability and reusability, enhancing performance across datasets within the same task, reducing computational overhead, and avoiding data dependency. moreover, the induced sparsity in attention heads helps mitigate knowledge forgetting in the model alignment. in summary, our contributions are as follows: ## methodology in this section, we first provide a preliminary overview of the attention mechanism with task parameters and then present alps in detail. fig 3 illustrates the overall framework. ## conclusion this work presents alps , an approach that leverages model weight parameters to localize and prune task-sensitive attention heads, leading to more efficient llm alignment while avoiding data dependency. extensive experiments on general, math, and code tasks demonstrate that our method consistently outperforms baselines, exhibits head transferability, and mitigates knowledge forgetting. overall, alps offers a promising perspective on efficient llm alignment and paves the way for further research into parameter-efficient alignment strategies that capitalize on the role of attention heads.