PILLOW : Enhancing Efficient Instruction Fine-tuning via Prompt Matching (2023)

Abstract: Instruction fine-tuning has conventionally been employed to adapt Large Language Models (LLMs) to a variety of diverse tasks. Nonetheless, this technique often necessitates substantial computational resources, making it impractical for deployment by individuals or small-scale entities. Recently, Low-Rank Adaptation (LoRA) has become a promising alternative, offering tuning capabilities with reduced resource overhead. However, attaining satisfactory performance through the fine-tuning of LoRA is a non-trivial challenge. In this paper, we propose PILLOW, which aims to improve LoRA’s performance by leveraging LLM’s in-context learning capability through prompt matching via reinforcement learning in resource-constrained environments. Specifically, PILLOW incorporates a matching network that selects prompts from a user-defined pool, concatenates the optimal prompts given the user instruction, and performs inference using the LoRA-fine-tuned LLMs. Compared with typical instruction fine-tuning methods, PILLOW exhibits commensurate performance on various evaluation metrics, utilizing only consumer-grade GPU resources and exhibiting a large increase in training efficiency.

## introduction in recent years, the impressive achievements of large language models (llms) have become increasingly evident. online llm products, e.g., claude @xcite and @xcite , have been widely recognized by the industry for their strong capabilities and are utilized in a myriad of industrial tasks @xcite . the achievement of such success highly hinges on the usage of supervised finetuning (sft) @xcite @xcite . nevertheless, as these models become larger, so does the intricacy of sft. these fine-tuning procedures typically demand a large scale of computational resources to accommodate training all the model parameters. consequently, this can be economically challenging for independent developers and smaller entities, who often have their own specific needs and budget limitations. in addition, data privacy standards prevent them from using third-party apis, adding another layer of constraint for them to utilize the llms. thus, while llms have been evolutionary in various applications, their scalability and cost-effectiveness still pose challenges in deployment. to solve the aforementioned problem, some have applied parameter-efficient finetuning which updates a relatively small portion of parameters, making fine-tuning more manageable under resource limitation @xcite @xcite @xcite . @xcite introduce lora to train dense layers by optimizing their rank decomposition matrices, thus considerably minimizing the number of trainable parameters and not adding to inference latency. however, llm's performance may be limited as lora only trains a subset of the model parameters. furthermore, lora may not achieve good performance on some tasks with unique characteristics because it can hardly adapt to diverse datasets due to its static fine-tuning strategy @xcite . therefore, can we attain a similar performance level to sft by merely employing a comparable amount of resources as used by lora? can we realize it using lora-fine-tuned llms' reserved in-context learning (icl) capacity? our approach, named pillow, trains a prompt matching net using reinforcement learning to improve finetuning llms under low-resource settings. we train an rl agent to select exemplars from a comprehensive "prompt set" which can be defined by question: based on the given title, create a summary of the paper. users or split from the training data, and subsequently merge these examples to form a prompt which is then added at the beginning of the input text that is fed into lora model. figure 1 shows an example where a 2-shot prompt is given by pil-low agent based on the user input. our approach becomes particularly beneficial in commercial applications where user input styles vastly deviate from those found in the pre-training corpus but are confined to limited variations. in these circumstances, the rl agent can efficiently learn to choose examples that best resonate with the specific query posed to the llms and therefore achieve comparable performance with direct sft training. our contributions can be summarized as follows: ## method we present pillow, a novel rl-based prompting matching framework, designed to enhance the performance of the fine-tuned lora model leveraging in-context learning. to provide a better understanding of our work, we first give a brief overview of the necessary background information. following that, we depict our task by framing it within the reinforcement learning settings and subsequently detail the components of our framework. ## experiments we conduct a comparative evaluation of our proposed framework pillow against two typical baseline methods: lora and sft. sft requires a high quantity of resources with high-quality response, while lora operates effectively under constrained resources but the performance is inferior to sft. nevertheless, empirical findings from our experimental studies suggest that pillow has the capability to yield performance in parity with sft, even under low-resource constraints. ## conclusion we train a prompt matching framework pillow via reinforcement learning to enhance efficient instruction finetuning. pillow is evaluated on the most recent instruction finetuning datasets, alpaca and dolly, and achieves superior results across all evaluation metrics and model sizes compared with supervised fine-tuning under lora. this new area of research combining prompting, matching, and rl can inspire future work on better prompting methods for llms under low-resource regimes.