H i F i: High-Information Attention Heads Hold for Parameter-Efficient Model Adaptation (2023)

Abstract: To fully leverage the advantages of large-scale pre-trained language models (PLMs) on downstream tasks, it has become a ubiquitous adaptation paradigm to fine-tune the entire parameters of PLMs. However, this paradigm poses issues of inefficient updating and resource over-consuming for fine-tuning in data-scarce and resource-limited scenarios, because of the large scale of parameters in PLMs. To alleviate these concerns, in this paper, we propose a parameter-efficient fine-tuning method HiFi, that is, only the highly informative and strongly correlated attention heads for the specific task are fine-tuned. To search for those significant attention heads, we develop a novel framework to analyze the effectiveness of heads. Specifically, we first model the relationship between heads into a graph from two perspectives of information richness and correlation, and then apply PageRank algorithm to determine the relative importance of each head. Extensive experiments on the GLUE benchmark demonstrate the effectiveness of our method, and show that HiFi obtains state-of-the-art performance over the prior baselines.

## introduction recently large-scale pre-trained language models (plms) have triggered a technological revolution in natural language processing (nlp), as the satisfactory performance could be achieved by fully finetuning parameters of plms @xcite @xcite . in datascarce and resource-limited scenarios, however, this methodology poses several concerns. on one hand, full fine-tuning leads to inefficient updating and catastrophic forgetting issues when the training set is insufficient @xcite ; on the other hand, this approach has to duplicate a modified copy of full parameters per for an example of a set of head weights, full fine-tuning updates all weights, whereas the non-structured methods randomly fine-tune a subset of parameters. for the structured methods, adapter-like models update the extra weights, while our proposed hifi selects several vital heads for fine-tuning. parameter-efficient fine-tuning (peft), as an alternative paradigm, has attracted more attention recently @xcite . compared to full fine-tuning, peft only fine-tunes the minority of the original parameters (or extra introduced parameters) instead of the entire parameters of plms, as shown in fig. 1 . although current pefts effectively decrease the proportion of trainable parameters, these methods also lead to varying aspects of concerns. for instance, adapter @xcite not only breaks the model structure by introducing additional parameters but also causes inference delays @xcite . we compare the representative models in tab. 1. motivated by these issues, in this paper, we propose hifi, a novel peft method by fine-tuning the relatively significant heads in multi-head attention (mha) modules, where we assume that our plms are based on transformer @xcite architecture and mha is chosen since it plays a crucial role in transformer according to recent studies @xcite @xcite . there are several intractable challenges to implementing our proposed method. how to measure the individual importance of a head? the core question of hifi is to select relative importance heads in mha. toward this end, we first analyze the importance of a single head. specifically, we decompose the output of each head by singular value decomposition (svd) to obtain a sequence of singular values in descending order, where if the cumulation of top-t terms account for a percentage threshold (e.g., 90%), the index t is treated as a measurement of information richness of the head. how to measure the relative importance between heads? given that the collaboration among multiple heads achieves success @xcite , the head-to-head correlation also shall be considered. therefore, we calculate the covariance matrix between the outputs of heads to measure the correlation across heads. to further work out those highly informative and strongly correlated heads, we model the relationship between heads into a graph based on their information richness and correlation, and then derive the relative importance of each head by pagerank algorithm @xcite . we illustrate our method overview in fig. 2 . to verify the effectiveness of our proposed method, we conduct extensive experiments on the glue benchmark @xcite , in both fullshot 1 and few-shot settings. the results show that our model hifi gains state-of-the-art performance against strong baselines. for example, full finetuning achieves the average score of 82.0% in the full-shot setting, while hifi obtains superior performance (82.3%). to summarize, our contributions are as follows: ## related work the existent research on peft can be generally divided into two folds: structured and non-structured methods. for the structured methods, the updated parameters are modularized (i.e., the parameters from the particular weight blocks are tuned), while the location of those updated parameters in the nonstructured methods is irregular, as shown in fig. 1 . ## conclusion in this paper, we propose a novel peft method hifi by modeling the relationship between attention heads as a graph and then using the pagerank to determine the relative significant heads for finetuning. hifi obtains state-of-the-art performance over the prior baselines on glue benchmark, in both full-shot and few-shot scenarios.