L ayer C onnect: Hypernetwork-Assisted Inter-Layer Connector to Enhance Parameter Efficiency (2022)

Abstract: Pre-trained Language Models (PLMs) are the cornerstone of the modern Natural Language Processing (NLP). However, as PLMs become heavier, fine tuning all their parameters loses their efficiency. Existing parameter-efficient methods generally focus on reducing the trainable parameters in PLMs but neglect the inference speed, which limits the ability to deploy PLMs. In this paper, we propose LayerConnect (hypernetwork-assisted inter-layer connectors) to enhance inference efficiency. Specifically, a light-weight connector with a linear structure is inserted between two Transformer layers, and the parameters inside each connector are tuned by a hypernetwork comprising an interpolator and a down-sampler. We perform extensive experiments on the widely used the GLUE benchmark. The experimental results verify the inference efficiency of our model. Compared to Adapter, our model parameters are reduced to approximately 11.75%, while the performance degradation is kept to less than 5% (2.5 points on average).

## introduction the emergence of pre-trained language models (plms) has brought natural language processing (nlp) to a new era @xcite . fine tuning all plms parameters has become the most common strategy to apply them to downstream tasks @xcite @xcite @xcite @xcite @xcite @xcite . however, in this manner, one should store a full copy of a plm for each downstream task because all its parameters are updated during the fine-tuning stage. thus, this strategy challenges the usage of the plms on edge devices (e.g., mobile phone and embedded systems) that have limited storage space. under this background, parameter efficiency has gradually attracted attention from both the research community and industries @xcite @xcite @xcite . the core concept of parameter efficiency is to reduce the trainable parameters of plms. consequently, when deploying a plm, only the different trained parameters from multiple tasks must be stored, together with one copy of the shared frozen parameters. currently, there are three main types of parameter efficiency methods: (1) specification-based methods train only a small part of the parameters in the original plm, while the others are frozen. for example, bitfit @xcite only updates the parameters of the bias and task-specific layers in plms. (2) prompt-based methods prepend additional context (i.e., prompts) to the original input, and change only the prompt parameters during fine tuning. in this manner, p-tuning @xcite adopts an extra network to functionalize the optimization of the continuous prompt embedding. then, prefix-tuning (li and liang, 2021) is proposed to use trainable prefixes to accomplish a parameter-efficiency task. (3) the adapter-based methods inject small-scale neural modules to plms and update only these modules during fine tuning. adapter @xcite simply injects two linearly trainable projectors (i.e., adapter layers) into each transformer layer of plms. karimi mahabadi et al. (2021) further make the weights of adapter layers conditioned on hypernetworks @xcite facilitating parameter sharing across multiple tasks. later, @xcite propose low-rank adaptation (lora), a layer-parallel structure which contains less parameters than adapter. in view of hypernetwork @xcite which generates weights for another network, although its superiority has been proved in multi-task parameter efficiency @xcite , the single linear projector used in their hypernetwork continues to have under-explored abilities (e.g., avoid-ing local optima and performance compensation when linear structure size is reduced). furthermore, the plm used in karimi mahabadi et al. ( 2021 ) is t5 @xcite , a pre-trained encoderdecoder model. therefore, the effectiveness of the hypernetwork remains unknown in encoder-only plms (e.g., bert and roberta). to this end, we propose layerconnect, a hypernetwork-assisted inter-layer connector. specifically, in the proposed architecture, we first insert a connector between every adjacent transformer layer in a plm. only the parameters of the inserted connector are updated during the fine-tuning stage. then, to share parameters across multiple tasks and avoid local convergence, the weights of the connector are conditioned on a well-designed hypernetwork that is equipped with an interpolator and a down-sampler. the proposed hypernetwork reduces the required parameters in the connectors and ensures sufficient optimization space. we conduct extensive experiments on the glue benchmark @xcite . the experimental results show that compared with adapter, our layerconnect requires only 11.75% of its parameters, while keeping the performance degradation to less than 5%. ## method we propose hypernetwork-assisted inter-layer connectors (i.e., layerconnect), as shown in figure 1 , the transformer encoder @xcite is used as the backbone of our layerconnect and a connector is inserted between transformer layers, named inter-layer connector. specifically, the connectors linearly transform the hidden states of each transformer layer: compared with previous intra-layer adapters @xcite , our inter-layer connector contains fewer trainable parameters due to (1) the number of inserted layers in our model is l while the counterpart of others is 2l; and (2) the transformer layer n connector downsampler + transformer structure … … … … inputs outputs transformer layer 2 transformer layer 1 connector to prevent the model from converging to the local optima due to the small number of parameters, the weights of a and b are conditioned on a hypernetwork. unlike the approach of karimi mahabadi et al. ( 2021 ), whose hypernetwork is a single linear projector, we design a new hypernetwork equipped with an interpolator and a down-sampler. in detail, we first utilize a learned task embedding e @xmath0, and then, use an interpolator to raise dimensionality, followed by a relu non-linearity: therefore, we only need to restore the connector parameters, i.e., a and b, for each task. ## discussion as presented in we compare the parameter size in the deployment for model inference as the number of tasks grows, as shown in figure 2 . in model inference, considering the scalability, for the bitfit, one may at least store multiple copies of the bias (i.e., 1,324k). however, the structure and the bias locations are also necessary. the parameter sizes of the adapter and the adapter with the hypernetwork are the same (i.e., the size of the bottleneck structure). in contrast, our connectors require an extremely small parameter size. that is to say, one can easily deploy several tens of the tasks, and the total size of the parameters just arrives at the same level as that of a single task when using the others. ## conclusion to enhance the parameter efficiency in the finetuning stage of plms, we propose the ultra-light connectors to be embedded into the transformer layers. furthermore, to keep the performance of such a small structure, we use the hypernetwork to assist the tuning of the parameters within the connectors. we compare our method with mainstream methods (adapter and bitfit). experimental results show that our method outperforms bitfit in most cases. compared with ada 8 (with best performance in most cases), our method reduces the trainable parameters to 11.75% for base models and 8.82% for large models, while keeping the performance degradation to less than 5% (2.07 points on average). by analyzing the results, we verify that reducing the number of parameters on the basis of adapter will seriously reduce the performance, and the introduction of hypernetwork promises an effective way to compensate for performance. this reveals a new direction of the study on model efficiency. additionally, layerconnect shows scalability, especially for the memory-sensitive and storage-sensitive edge devices e.g., smartphones, embedded devices, micro containers, and iot/ioh devices.