C on T in T in: Continual Learning from Task Instructions (2022)

Abstract: The mainstream machine learning paradigms for NLP often work with two underlying presumptions. First, the target task is predefined and static; a system merely needs to learn to solve it exclusively. Second, the supervision of a task mainly comes from a set of labeled examples. A question arises: how to build a system that can keep learning new tasks from their instructions?This work defines a new learning paradigm ConTinTin (Continual Learning from Task Instructions), in which a system should learn a sequence of new tasks one by one, each task is explained by a piece of textual instruction. The system is required to (i) generate the expected outputs of a new task by learning from its instruction, (ii) transfer the knowledge acquired from upstream tasks to help solve downstream tasks (i.e., forward-transfer), and (iii) retain or even improve the performance on earlier tasks after learning new tasks (i.e., backward-transfer). This new problem is studied on a stream of more than 60 tasks, each equipped with an instruction. Technically, our method InstructionSpeak contains two strategies that make full use of task instructions to improve forward-transfer and backward-transfer: one is to learn from negative outputs, the other is to re-visit instructions of previous tasks. To our knowledge, this is the first time to study ConTinTin in NLP. In addition to the problem formulation and our promising approach, this work also contributes to providing rich analyses for the community to better understand this novel learning problem.

## introduction the main goal of machine learning algorithms lies in seeking supervision for solving a target task. traditionally, the supervision is extracted from a set of labeled examples. the learner constructs a decision function that generalizes beyond the seen examples. ## related work this section retrospects continual learning and learning from task instructions, two machine learning paradigms that try to explore supervisions s 1 and s 2 , respectively. continual learning. since the advent of continual learning 2 @xcite , this learn-1 "transferring distance" refers to the task numbers between the model at a new status and the model at an earlier status. ccl learns a sequence of classes (e.g., visual object categories, text labels, etc.) to build one overall multi-label classifier for all the classes seen so far @xcite . for example, wang et al. (2019) dcl essentially studies the same task but in different domains. the system is expected to evolve along with learning from a stream of datasets of the same task and different data distributions. typical work in nlp includes sentiment classification @xcite , conversational agents @xcite , text classification, and question answering @xcite , etc. tcl tries to learn distinct tasks sequentially. systems in @xcite incrementally learned among five disparate nlp tasks. @xcite further extended the size of the task stream (one benchmark has 26 tasks, the other covers 55) and studied tcl in a few-shot scenario. it is worth mentioning that all the listed work in tcl consistently transformed all tasks into question answering format (as pointed out in @xcite , many nlp tasks can be formulated as question answering), thus tcl in these literature was actually converted into dcl. similar with @xcite , our work also focuses on low-resource continual learning; in contrast, our learning problem belongs and cohen, 1989), and never-ending learning @xcite . to tcl while each task in our formulation is expressed by instructions instead of labeled examples. learning from textual instructions. this learning paradigm was first presented by @xcite . they investigated the challenges on solitaire card game where an instruction is a short sentence such as "you can move any top card to a free cell if it is empty", then this instruction is mapped into logical expression via semantic parsing so that an automated agent can understand and execute the instruction. more recent work tried to examine the ability of large-scale pretrained language models to follow natural language instructions of varying complexity. for example, efrat and levy (2020) tested gpt-2 @xcite to understand instructions like "listing nouns", "output the nth word or char" and real-world mturk instructions to annotate some popular datasets. they concluded that gpt-2 works poorly when the supervision comes from those instructions. a dominant instruction format nowadays is called "prompt" which mostly is a short piece of text describing the core concept of the task. representative work includes @xcite @xcite , etc. (please refer to the survey @xcite for more details.) while these prompt-based results are encouraging, such prompts are often too simplistic, whereas many real nlp problems cannot be effectively formulated as short prompts or a few positive examples. motivated, @xcite collected more than 60 distinct nlp tasks with realworld mturk instructions, and claimed that pretrained language models, such as bart and @xcite , benefit from instructions to generalize across tasks. to our knowledge, the only work somehow resembling ours is @xcite , in which task descriptions were incorporated into lifelong learning for zero-shot transfer. we differ in three aspects: (i) they focused on robot controlling problems, (ii) their tasks are from a single domain, and (iii) in addition to the associated instruction, they assumed that each task has a large number of labeled examples. ## our method instructionspeak most prior studies about continual learning focused on backward-transfer @xcite while paying less attention to the forward-transfer performance. next, we introduce our approach to promoting both of them. the big story of our strategies lies in better understanding of the textual instruction of u i . two concrete strategies as follows. negative training: to distinguish favorable and unfavorable clues in instructions. unfavorable clues, such as the red items in figure 2 , are essential for humans to make decisions while not being successfully leveraged by machine learning. for example, @xcite found discarding negative examples can even improve the performance. we believe this indicates the approach failed to learn from negative examples rather than those examples being truly useless. then, how can we make machines extract effective supervision from negative samples? first, we introduce a method that was tried but did not work well -minimizing the probability of generating negative output. maximizing the probabilities of gold output is widely used in text generation. it sounds intuitive to minimize that for unwanted output, such as @xcite . we tried a joint training on maximizing positive and minimizing negative examples, which is even worse than maximizing the positive alone. since many "negative" outputs contain tokens that exist in the gold answers, we suspect that minimizing their probabilities will let the model have more difficulty decoding the correct output. after further study of those negative examples and their explanations, we decide to treat those negative examples as positive and move the negative learning phase as pretraining, i.e., pretrain on negative examples first, then finetune on positive examples. the inspiration comes from the fact that negative examples, despite the tag "negative", can still provide useful information about the expected output. take a look at the negative example in figure 2, its output "c i.e., color and shape of the rock" is discouraged just because it does not follow some rules of automatic evaluation rather than it is really wrong. apparently, as a first step, optimizing the system to generate the so-called "negative output" is still better than any general-purpose pretrained bart. for each unseen task in u , we directly adopt its negative examples if available. for the k training tasks in s, positive instances (including positive examples in instructions and those labeled task instances) are much more than the negative examples, we use the pretrained model on s to do prediction on all inputs of s, if the output is not equal to the gold output, we treat this (input, predicted output) as a negative example. it means we have a loose definition of what "negative output" is: it is negative once it is not equal to the ground truth. since the pretrained model on s can already guarantee generation quality, those generated negative outputs are mostly related with the gold outputs (measured by rouge metrics). 3 history training: revisit instructions of previous tasks. to mitigate catastrophic forgetting, many prior works about continual learning tried to store a couple of labeled examples of upstreaming tasks to replay. in our contintin formulation, each new task is described merely by the instruction. instead of storing some examples of previous tasks, we keep their instructions. when learning the ith task in u , our model will first learn all the instructions of prior i-2 tasks in a batch with a lower learning rate. revisiting precedent instructions is cost-effective since each instruction is as short as a couple of conventionally annotated examples but with much more supervision. overall, our two strategies work jointly to enhance the forward-transfer and the backward-transfer performance. our system instructionspeak is based on bart, treating all tasks as a text-to-text problem. the full input format of encoder: ## experiments setup. we use the pretrained bart-base model released by huggingface. hyperparameters: @xmath0; @xmath1; max input length 1024 tokens, learning rate 5e-5, 3 epochs as suggested by @xcite for most phases of training (except for 5e-6 and one epoch for history training); batch size 5 for training on s and 2 for continual learning on u . all unseen tasks u randomly select 1k labeled examples for performance evaluation. note that the official evaluation metric for natural-instructions is rouge-l @xcite . according to the definitions of our evaluation metrics, -→ g i and ←g i numbers are the same meaning as ## conclusions this work introduced a novel learning problem: continual learning from task instructions. the goal is to explore the potential of exiting pretrained language models in solving new tasks by understanding instructions rather than labeled examples. with our problem formulation and a well-performing system, we pave the way for future study of this challenge in the community.