Empowering parameter-efficient transfer learning by recognizing the kernel structure in self-attention (2022)

Abstract: The massive amount of trainable parameters in the pre-trained language models (PLMs) makes them hard to be deployed to multiple downstream tasks. To address this issue, parameter-efficient transfer learning methods have been proposed to tune only a few parameters during fine-tuning while freezing the rest. This paper looks at existing methods along this line through the kernel lens . Motivated by the connection between self-attention in transformer-based PLMs and kernel learning, we propose kernel-wise adapters , namely Kernel-mix , that utilize the kernel structure in self-attention to guide the assignment of the tunable parameters. These adapters use guidelines found in classical kernel learning and enable separate parameter tuning for each attention head. Our empirical results, over a diverse set of natural language generation and understanding tasks, show that our proposed adapters can attain or improve the strong performance of existing baselines.

## introduction transfer learning using large-scale transformerbased pre-trained language models (plms) @xcite has become the standard scheme for various natural language processing (nlp) tasks. among many strategies, fine-tuning these plms emerges as the predominant strategy to adapt the generic models to a specific task @xcite . however, deploying these models is a challenge as curating customized models across a wide variety of tasks would lead to scalability issues. it requires one to store (and sometimes move) multiple copies of the plm parameters for different tasks, which is inefficient. a popular approach to tackling such scalability issues is to make the plm-based transfer learning more parameter-efficient. this can be done by freezing most of the plm parameters and inserting small trainable modules into the plm. adapters @xcite @xcite and prefix-/prompt-tuning @xcite @xcite have emerged as the prominent approaches under this paradigm. these methods are incredibly parameterefficient and have comparable performance to full fine-tuning models on many common nlp tasks (mainly in natural language understanding) by tuning only 0.1-3% task-specific parameters of the original plms. however, most of these studies take the plms as a black box, i.e., these methods are not customized to transformers. this raises whether parameter-efficient fine-tuning has fully utilized the transformer structure in plms. therefore in this work, we propose kernel-wise adaptation, which recognizes and utilizes the kernel structure within self-attention-the core component in a transformer. specifically, we take inspiration from recent work that connects self-attention to kernel learning @xcite @xcite to treat the different attention heads in a transformer's attention sub-layer as separate kernel estimators. we hypothesize that parameter-efficient tuning can benefit from some useful guidelines in classical kernel learning literature and incorporate them into our proposed methods. these include: we discuss these guidelines in detail in § 4.1. we also evaluate our hypotheses through rigorous empirical evaluation. first, we test the effectiveness of the two guidelines above by comparing the default lora @xcite -a state-of-the-art approach for efficient adaptation-and two of its variants that implement the two guidelines, respectively. next, we evaluate our variant of kernel-wise adaptation on three natural language generation (nlg) benchmarks and two natural language understanding (nlu) tasks using the gpt-2 architecture. while parameter-efficient work has extensively covered nlu tasks, it is unknown how well the results transfer to nlg tasks. as language generation typically requires more expressive models, we put more emphasis on multiple nlg tasks that include data-to-text, free-form question answering (qa), and summarization. the empirical results in § 6 demonstrate that with the same parameter budgets, our proposed method can attain better generation quality and classification accuracy than previous techniques, and in many settings, it is close to or even outperforms the full parameter fine-tuning. ## related work the literature on parameter-efficient adaptation can be broadly categorized as follows: adapters. originally proposed by @xcite ; @xcite , adapters modulate the output of a transformer layer by inserting small multi-layer perception (mlp) bottleneck layers. recent work has proposed many variants of the original adapters, including dropping adapters across several layers @xcite or constraining adapters to be low-rank operators @xcite . a recent line of work focused on identifying the important subset of parameters within the plms. ben @xcite proposed to only tune the bias terms in the plms. mpop @xcite suggested decomposing the weight matrices in plms through matrix product operators (mpo) and only trained the matrices of small size (freezing the large matrices) obtained from the decomposition, which implicitly recognizes the small matrices as the important subset. low-rank adaptation (lora) (hu et al., 2021) directly assumes that the update of the weight matrices during training can be approximate low-rank, and accordingly proposed to re-parameterize the original weight matrix w by w + ba, where w is frozen to its pre-trained weights whereas @xmath0, @xmath1. we note that lora too introduces new weights a and b similar to adapters, but they are used only to re-parameterize the existing weights and do not add extra sandwiched layers that modify the original model architecture. ## preliminaries we start by providing a brief introduction to the transformer architecture ( § 3.1) and then revisit the connection between attention and kernel estimators ( § 3.2), building on which we propose the kernelwise adapter in § 4. ## method we introduce our kernel-mix method in this section, which builds upon the proposed kernel-wise adaptation. to explain the principle behind kernelwise, we first illustrate the guidelines we aim to adopt in our adapter design and show that existing methods fail to satisfy them ( § 4.1). with these details, we finally propose our method in § 4.2 and § 4.3. ## experiments while parameter-efficient tuning methods have been extensively studied for nlu tasks, their applicability towards nlg tasks is not well-known. this section performs empirical experiments of our proposed methods on three nlg tasks. to show the consistent effectiveness of our methods, we provide results on two nlu tasks as well. foot_7 ## conclusion and future work in this work, we revisit the connection between the attention module and kernel estimators, and accordingly propose kernel-wise adaptation, which adopts the guidelines from kernel learning to strengthen the low-rank adaptation (lora). we verify that with the same parameter budgets, our proposed adaptation techniques can have better performance on three generation tasks than the existing parameter-efficient methods, including adapters, prefix-tuning, and lora, and attain close accuracy on two classification tasks as well. one possible extension of our work is combining our proposed method with other adapters in feed-forward sub-layers. in mam-adapter, @xcite suggest applying prefix-tuning to adapt the parameters in self-attention sub-layers and assigning budgets to feed-forward sublayers as well; it can be beneficial to replace prefix-tuning with kernel-mix for adaptation in the attention part. another direction of future research is the extension of our method to the feed-forward sub-layers, which are interpreted as key-value memories in recent work and behave like attention blocks @xcite . it will be interesting to study if kernel-specific guidelines could help design better adapters employed in the feed-forward layers.