Large-scale Lifelong Learning of In-context Instructions and How to Tackle It (2023)

Abstract: Jointly fine-tuning a Pre-trained Language Model (PLM) on a pre-defined set of tasks with in-context instructions has been proven to improve its generalization performance, allowing us to build a universal language model that can be deployed across task boundaries. In this work, we explore for the first time whether this attractive property of in-context instruction learning can be extended to a scenario in which tasks are fed to the target PLM in a sequential manner. The primary objective of so-called lifelong in-context instruction learning is to improve the target PLM’s instance- and task-level generalization performance as it observes more tasks. DynaInst, the proposed method to lifelong in-context instruction learning, achieves noticeable improvements in both types of generalization, nearly reaching the upper bound performance obtained through joint training.

## introduction a number of recent studies have shown that pretrained language models (plms) only need to undergo a brief fine-tuning process to achieve remarkable instance-level generalization performance within the observed task @xcite . compared to instance-level generalization within seen tasks, however, the zero-shot cross-task generalization capability of plms to unseen tasks is just starting to be explored @xcite . in order to build a language model that can generalize across task boundaries and thus be deployed to various task scenarios, in-context instruction learning draws inspiration from how humans can familiarize themselves with a variety of language-related tasks only by following instructions @xcite . to emulate the human learning process, in-context instruction learning trains the target plm with both input-output pairs and a set of in-context instructions that contain additional task-specific information. the introduction of in-context instructions is spearheading a noticeable progress in cross-task generalization within natural language processing (nlp) research. the recent advances in-context instruciton learning @xcite @xcite indicate that finetuning a plm with task-specific instructions has a positive impact on its cross-task generalization capability. one common limitation shared by the existing works is that they require a static, pre-defined set of tasks to jointly train the target plm. any learning paradigm bound by the assumption that all of the tasks are pre-defined and non-changing not only incurs a huge memory and computational cost but also raises serious data privacy concerns @xcite . in this paper, we aim to address such concerns by studying whether it is possible to sequentially fine-tune the target plm on a stream of large-scale instruction-paired tasks in a lifelong manner. in the same spirit as joint in-context instruction learning, the primary objective of lifelong in-context instruction learning is to gradually improve both instance-and task-level generalization capabilities as the target plm observes more train tasks. with the help of lifelong in-context instruction learning, deployment of a universal language model on edge devices with limited memory and computation power becomes easier. in such a resource-constrained setting, it is infeasible to jointly train a language model from scratch on all user data every time a new task is introduced. moreover, it is difficult to deploy a separate model for every task or utilize an extremely large model with multi-task capability. instead, we can deploy a continuously evolving language model and train it sequentially on a task stream. to study the problem of lifelong in-context instruction learning within the context of instance-and task-level generalization, we adopt super-naturalinstructions (sup-natinst) @xcite , which is the largest dataset with in-context instructions to date, and restructure it accordingly. because we believe that cross-language generalization is beyond the scope of this paper, we use the english subset of sup-natinst, and from here on, we use sup-natinst to refer to the english subset instead of the entire dataset. more details on the characteristics of sup-natinst and the data restructuring process are provided in section 3. our proposed approach to lifelong in-context instruction learning, dynainst, combines parameter regularization and experience replay. the regularizer employed by dynainst is designed to induce wide local minima in the target plm. deep neural networks with wide local minima are known to achieve improved generalization performance and become more robust against task distribution shifts @xcite ; these two advantages of wide local minima are well-aligned with the objectives of lifelong instruction learning, making it a particularly attractive choice of regularization. to design a memory-and compute-efficient experience replay framework, we devise dynamic instruction replay, which is comprised of dynamic instance selection (dis) and dynamic task selection (dts). dis and dts flexibly determine which instances and tasks are stored and replayed, respectively. our experimental results demonstrate that dynainst outperforms strong baselines in both instance-and task-level generalization under various experimental scenarios. our contributions can be summarized as follows: ## problem definition in this section, we describe sup-natinst and how it is restructured for the purpose of lifelong in-context instruction learning. then, we formulate evaluation metrics for quantifying the instance-and task-level generalization capabilities. the visual summary of problem definition is provided in figure 1 . ## methodology we introduce dynainst, our approach to lifelong in-context instruction learning. dynainst is a hybrid method that combines parameter regularization and experience replay. in section 4.1, we elaborate on the use of wide local minima-inducing regularizer for lifelong instruction learning. then, in section 4.2, we describe how instances and tasks are dynamically stored and replayed through dynamic instruction replay. the lifelong instruction learning process with dynainst is illustrated in figure 2 . we also provide line-by-line description of dynainst in algorithm 1. ## ablation studies 6.1 separate components to validate the efficacy of each technical component in dynainst, we perform a component-wise analysis of dynainst and report the results in ## conclusion in this work, a fully lifelong learning of in-context instructions was investigated for the first time. we proposed dynainst, a novel hybrid approach to lifelong in-context instruction learning, and verified its superiority to existing baselines under various experimental scenarios. potential directions for future research include extending our investigation to blurred or unknown task boundaries and analyzing whether dynainst outputs biased predictions.