Toward Zero-Shot Instruction Following (2024)

Abstract: This work proposes a challenging yet more realistic setting for zero-shot cross-task generalization: zero-shot instruction following, presuming the existence of a paragraph-style task definition while no demonstrations exist. To better learn the task supervision from the definition, we propose two strategies: first, to automatically find out the critical sentences in the definition; second, a ranking objective to force the model to generate the gold outputs with higher probabilities when those critical parts are highlighted in the definition. The joint efforts of the two strategies yield state-of-the-art performance on the Super-NaturalInstructions. Our code is available on GitHub.

## introduction with the rapid evolutions of the pre-training techniques, large language models (llms), such as gpt-3 @xcite and @xcite , are found to be capable of handling various novel nlp tasks by following in-context instructions @xcite . 2 typically, a formal task instruction consists of two components: (1) a task definition that describes the task intent; (2) a few labeled examples to demonstrate this intent (i.e., demonstrations). then the problem is often named as "k-shot instruction following", where k is the example size. due to the performance superiority of the in-context examples @xcite , prior research has predominantly relied on demonstrations, allocating relatively limited attention toward effectively utilizing task definitions; we refer to this setting as "demonstration-driven instruction following" @xcite . notwithstanding the surprising results, this phenomenon could manifest as an instance of overestimated progress. two reasons: firstly, demonstrations are usually hard to be crafted in real-world applications. since llms are becoming helpful daily-task assistants and most end-users are nonexperts @xcite @xcite , it is usually exhausting and unrealistic for users to design concrete demonstrations for every daily task, especially for those tasks that require specific domain knowledge. secondly, as @xcite concluded, so far, we still lack a method to effectively learn from instructions to solve tasks without demonstrations for various reasons. for example, @xcite showed that the models constantly ignored the crucial information emphasized in the definition (e.g., an output constraint that asks models to "generate no more than five words"); @xcite found that the models always struggled to truly understand the content of the definition. to more effectively utilize the task definition, this work studies a more challenging setting: zeroshot instruction following. technically, our approach consists of two strategies. 3 (i) strategy i: automatically learn the critical task-relevant information from the lengthy definition to help the model better grasp the instruction. (ii) strategy ii: to make the model truly distinguish instructions that are specified by the critical information or not, we set a ranking-based training objective. given instructions with critical information highlighted, this ranking strategy forces the model to generate ground-truth outputs with higher probabilities than instructions otherwise. our system, pick&rank, achieves state-of-the-art on the benchmark, super-naturalinstru @xcite . strategy i aims to predict a binary value for each sentence in a definition, indicating whether a sentence is crucial. the outputs of strategy i are used to construct instructions with different sufficiencies, e.g., "repeat" represents the most beneficial instructions where the crucial sentences are repeated. strategy ii then drives the lms to generate higher ground-truth probabilities on the more beneficial instructions. the whole system is optimized end-to-end. ## related work prompt & in-context learning. prompting techniques usually acquire answers from large language models (llms) after rewriting the original task input into a llm-oriented format. impressive progress has been observed in various nlp tasks, such as question answering @xcite ), text generation @xcite , information extraction @xcite , etc. @xcite further developed in-context learning (icl): gpt-3 achieved competitive few-shot results without parameter tuning by prepending a prompt with a few demonstrations to new inputs. follow-up work delved into improving icl, including how to choose better demonstrations @xcite , how to formulate the tasks @xcite , etc. however, the short and simplistic nature of the prompts makes it difficult to express nlp tasks of diverse complexities @xcite . our work tries to learn from instructions that describe the task semantics in more detail, such as amazon mturk instructions. ## problem definition & our approach we study zero-shot instruction following in a crosstask generalization setting, where evaluation tasks are unseen in training. zero-shot instruction following: three task sets: training tasks, dev tasks, and test tasks. there are no overlapping tasks among them. each task t has its instruction i and a collection of labeled examples @xmath0, y)}. x: input; y: gold output of x under i. i is a short paragraph consisting of n sentences, i.e., @xmath1, s 2 , • • • , s n }. no examples exist in i. d of dev tasks and test tasks are only used for evaluation. as shown in figure 1 , we adopt two strategies to better leverage the supervision in i. ## experiments dataset. we work on the benchmark super-naturalinstru @xcite , which contains 1,040 diverse english tasks (921 in train and 119 unseen tasks in test). we follow @xcite only using 756 tasks in train to train the final model. each task is expressed by an instruction, originally consisting of a paragraph-level task definition and a couple of positive&negative examples, and a large set of input-output instances. to satisfy our setting, we only use definitions as instruction i. the average definition length is 65.73 by words (4.09 by sentences). those classification and generation tasks are respectively evaluated by @xcite . we also report rouge-l (overall), which calculates the rouge-l on both classification and generation tasks, to reflect an overall estimation. more dataset and metric details can be found in appendix and baselines. since prior systems for few-shot instruction following need examples in instructions, in order to apply them to a zero-shot setting, we @xcite . more details about baselines are in the appendix. our model implementation. we follow wang et al. (2022b) using t5-base @xcite for all experiments. please refer to appendix and results. i: the answer will be "yes" if the provided sentence contains an explicit mention that answers the given question. otherwise, the answer should be "no". instances where the answer is implied from the sentence using "instinct" or "common sense for metaicl, due to the huge task differences between train and test (as shown in analysis. we try to clear up three concerns. i: generate an overlapping word between the given two sentences. [• • • ] you must generate significant words which are not the stop words like "the" or "of", etc. c : ?" [• • • ] "a : b" relates item a to whether it is trash or treasure, as specified by b. [• • • ] x: baby : treasure. leaf : ? y: trash || ŷ: relates item a to whether it is trash or treasure error type: pattern copy i: [• • • ] if it is about requesting something, generate 'request'. [• • • ] if it is about informing something, generate "inform". x: please tell me do you have any particular date for the event? y: request || ŷ: inform error type: incomplete critical sent. detection q 1 : did the detected critical sentences really contribute to the generation of gold outputs? to answer q 1 , we checked some examples where our system improves over the strongest baseline tk-instruct. as shown in q 2 : could ranking objective really improve the probability of gold outputs? regarding q 2 , we test our model on all test tasks with two versions of task instructions: repeat vs. origin. for each version, we calculate the corresponding probability of the ground truth output by averaging token-level probabilities in the output string. our model can produce a higher ground-truth probability once "repeat" instruction is adopted (score: 0.59) than the "origin" definition (score: 0.11), 6 demonstrating the effectiveness of our strategy ii. q 3 : error patterns of our systems. we randomly pick up 200 instances from the test and summarize three main error patterns of pick&rank, as shown in to detect the sentence that has a specific requirement "generate significant words which are not the stop words • • • ", the negation "are not" was not successfully comprehended by the system. unfortunately, negation understanding has increasingly been a challenge in nlp (al-khawaldeh, 2019; @xcite . (ii) pattern copy. the second example shows the system sometimes copies a span from the definition, especially when the definition string, e.g., "'a : b' relates item a to whether it is trash or treasure, as specified by b.", matches the format of x, e.g., "baby : treasure. leaf: ?". this resembles demonstration-driven incontext learning, where researchers found pattern match is a key factor of success @xcite . (iii) incomplete critical sentence detection. it is possible that our system detects partial sentences that are critical. as a result, the system is biased toward the requirement of highlighted sentences. rather than using a hard masking scheme, our future work will explore a soft-masking technique so that no instruction parts will be clearly ignored. ## conclusion in this paper, we focused on zero-shot instruction following, where we only adopted the task definitions as the instructions to help the model perform cross-task generalization. expressly, our method pointed the critical sentences out of the lengthy definitions and highlighted them explicitly. in addition, we further designed a ranking objective to improve the instruction grasp of the lms. we also conducted thorough analyses to help future research on zero-shot instruction following. here, "lr" denotes the learning rate; α is the probability margin in equation 3, there are three different α according to three ranking losses; β is a coefficient that controls the influence of the ranking losses; and k is the sampling times in equation 2.