Prefix Propagation: Parameter-Efficient Tuning for Long Sequences (2023)

Abstract: Parameter-efficient tuning aims to mitigate the large memory requirements of adapting pretrained language models for downstream tasks. For example, one popular method, prefix-tuning, prepends trainable tokens to sequences while freezing the rest of the model’s parameters. Although such models attain comparable performance with fine-tuning when applied to sequences with short to moderate lengths, we show their inferior performance when modelling long sequences. To bridge this gap, we propose prefix-propagation, a simple but effective approach that conditions prefixes on previous hidden states. We empirically demonstrate that prefix-propagation outperforms prefix-tuning across long-document tasks, while using 50% fewer parameters. To further investigate the proposed architecture, we also show its advantage in calibration, and perform additional study on its relationship with kernel attention. To the best of our knowledge, this work is the first to focus on parameter-efficient learning for long-sequence language tasks.

## introduction the transformer architecture @xcite has changed the landscape of recent natural language processing approaches by enabling the pretraining of state-of-the-art large language models (llm) @xcite @xcite . however, fine-tuning and storing full copies of llms can consume prohibitively large quantities of resources. parameter-efficient finetuning (peft) methods such as prefix-tuning @xcite @xcite address these concerns by reducing the number * work done during a student internship at ingenuity labs. in this paper, we propose a simple and effective method, prefix-propagation, which consistently improves the performance of peft for long sequence models. unlike prefix-tuning, prefix-propagation propagates the hidden states corresponding to prefixes through the attention computation. this allows for the prefixes hidden states to dynamically change as the input propagates through each layer. to further understand prefix propagation, we investigate the reliability of the model's predictions by performing analyses on calibration. lastly, we conduct study on prefix-based methods in terms of kernel attention to strengthen their theoretical value. in summary, our contributions are as follows: parameter-efficient tuning inspired by the success of manual prompting @xcite @xcite prepends trainable "soft" prompts to an input sequence. although further peft methods have since been introduced @xcite @xcite , we focus on adapting prefix-tuning. we note that our adaptation does not violate orthogonality and thus prefixpropagation can still be compounded with other peft methods as proposed in the unifiedpet framework @xcite , likely yielding similar performance gains. we leave the empirical validation of this hypothesis for future work. out work also adheres to the key motivation of the recent peft method, inducer-tuning @xcite , which is that optimal prefixes should be close to queries within their latent space. we derive queries, keys, and values from the same prefix token, limiting the distance that separates them. ## experiments and results datasets we evaluate our approach on three longdocument classification tasks: arxiv @xcite , an 11-class classification task composed of academic research papers, the 20-newsgroups @xcite classification task consisting of mailing lists that fall into one of 20 classes, and the hyperpartisan dataset, a binary classification task for extremist news classification @xcite . we also run experiments on wikihop @xcite , a long-document reading comprehension task requiring multi-step reasoning. due to compute limitations inherent to working with long documents, with the exception of hyperpartisan, we only report a single run for each task. this mimics the original longformer reporting scheme @xcite . for hyperpartisan, the smallest of the datasets, we report mean metrics averaged over five seeds. baselines as a baseline, we fine-tune longformer-base (approx. ## conclusion our research focuses on parameter efficient tuning for long documents tasks. we introduce prefix-propagation, which consistently improves performance over prefix-turning on long document datasets, while using 50% fewer parameters. we study the reliability of the predictions by performing analyses on calibration and show that models tuned with prefix-propagation are better calibrated. we lastly explicate prefix-propagation from a kernel perspective, uncovering insights for future peft research.