H yper L o RA : Efficient Cross-task Generalization via Constrained Low-Rank Adapters Generation (2024)

Abstract: Adapting pre-trained language models (PLMs) for cross-task generalization is a crucial research area within the field of NLP. While fine-tuning and in-context learning are effective approaches for adapting LMs to emerging tasks, they can be costly and inefficient. Recently, some researchers have focused on achieving efficient task adaptation via hypernetwork, which is a meta network that generates task-specific weights based on task-oriented information without any optimization. However, the training of hypernetworks often lacks stability since the optimization signal is not straightforward, and the task information is not adequately representative. Moreover, previous works train hypenetworks with the general corpus, which is struggling with few-shot adaptation. To address these issues, we introduce HyperLoRA, a hypernetwork for LoRA parameters generation involving hypernetwork pre-training on instruction-following data and generalization fine-tuning on sparse task data. Furthermore, we utilize a constrained training loss and a gradient-based demonstration selection strategy to enhance the training stability and performance. Experimental results and analysis across four benchmark datasets (P3, S-NI, BBH, and SuperGLUE) demonstrate the proposed approach has flexible generalization ability and superior performance.

## introduction pre-trained language models (plms) have shown remarkable capabilities across a diverse spectrum of nlp tasks, encompassing understanding @xcite , reasoning @xcite , and generation @xcite . the ability of language models to effectively adapt their knowledge to unseen tasks (referred to cross-task generalization) is crucial for the broader applicability of nlp systems, which garnered significant attention from many researchers. there are several approaches towards achieving cross-task generalization. the most straightforward way is fine-tuning lms with a certain amount of task-specific data, which demands substantial computational costs and may cause catastrophic forgetting, degenerating the performance of lms on the previous tasks @xcite . in contrast, in-context learning (icl) provides a few demonstration examples to generalize lms to unseen tasks without explicit optimization @xcite . however, icl requires extremely long and expensive-to-process inputs for each test example, making it both costly and inefficient @xcite . another line of researchers explores the cheaper and more effective approach that composes the weights of new tasks by selecting and combining fine-tuned or parameter-efficient weights from a pre-existing weights pool @xcite @xcite . while this approach is simple and effective, it necessitates a pre-existing pool of various task weights, and the effectiveness of the composed weights may be restricted by the pool of available tasks. additionally, although the parameter-efficient weights are lightweight, there are still resource consumption issues for storage and training while the pre-existing pool is substantial in various task scenarios. to address the above issues in the cross-task generalization scenario, @xcite @xcite proposes a meta network named hypernetwork, which performs a "text-to-weight" task converting task information (e.g. task instructions and task demonstrations) into task-specific parameters (e.g. prefixes @xcite , adapters @xcite , lora @xcite ) for underlying pre-trained language models. compared with fine-tuning and icl, hypernetwork is a more efficient method that generates task-specific parameters in a single forward without any optimization. furthermore, hypernetwork can generate parameters when required, avoiding additional storage expenditures. despite the obvious strength of previous works on hypernetwork, several issues remain that have not been appropriately solved. firstly, the training instability problem is one of the most challenging problems in hypernetwork training @xcite . existing strategies primarily rely on backpropagating gradients from the underlying model, while lacking effective and specific measures to circumvent this instability. secondly, most current works construct task demonstrations through manual crafting or random sampling from datasets, potentially affecting parameter generation and resulting in suboptimal performance. thirdly, previous works typically pre-train hypernetworks on general language corpus (e.g. c4 @xcite ) directly, which may not be adept at handling diverse task instructions and could perform suboptimally on low-resource task data. considering all the above considerations, we introduce hyperlora, a novel method that aims to enable language models for efficient cross-task generalization. specifically, hyperlora consists of a text encoder and a p-generator, designed to convert task information into parameter-efficient modules, specifically lora @xcite . to improve training stability, we propose an explicit training loss to constraint the training of hyperlora and utilize a gradient-based automatic demonstration selection strategy to select the most representative task examples. furthermore, our hyperlora involves hypernetwork pre-training on instructionfollowing data to enable it to generate task-related parameters based on task information and then generalization fine-tuning on sparse task data to adapt lms with unseen tasks. in a nutshell, the contributions of our work are as follows: furthermore, we introduce a constrained training objective to enhance training stability and develop an automatic demonstration selection strategy to further improve its performance. ## methodology in general, ∆w is scaled by α r , where α is a constant in r. ## conclusion in this paper, we propose hyperlora, a hypernetwork that generates efficient parameters for crosstask generalization. compared with in-context learning and peft, our method is more efficient which decreases the training and storage costs. furthermore, we propose a paradigm involving multitask instruction pre-training and generalization finetuning for hypernetworks. through comprehensive experiments and analysis on four benchmark datasets, we have shown hyperlora achieves better results than a series of multi-task learning and hypernetwork-based methods. in future, we plan to extend the proposed approach to cross-lingual and cross-modal generalization scenarios and explore the underlying models with larger scales.