LLM -driven Instruction Following: Progresses and Concerns (2023)

Abstract: The progress of natural language processing (NLP) is primarily driven by machine learning that optimizes a system on a large-scale set of task-specific labeled examples. This learning paradigm limits the ability of machines to have the same capabilities as humans in handling new tasks since humans can often solve unseen tasks with a couple of examples accompanied by task instructions. In addition, we may not have a chance to prepare task-specific examples of large-volume for new tasks because we cannot foresee what task needs to be addressed next and how complex to annotate for it. Therefore, task instructions act as a novel and promising resource for supervision. This tutorial targets researchers and practitioners who are interested in AI and ML technologies for NLP generalization in a low-shot scenario. In particular, we will present a diverse thread of instruction-driven NLP studies that try to answer the following questions: (i) What is task instruction? (ii) How is the process of creating datasets and evaluating systems conducted? (iii) How to encode task instructions? (iv) When and why do some instructions work better? (v) What concerns remain in LLM-driven instruction following? We will discuss several lines of frontier research that tackle those challenges and will conclude the tutorial by outlining directions for further investigation.

## introduction this proposal is driven by a fundamental question of task generalization in nlp: how to comprehend a new task if labeled examples are pretty limited? one goal of ai is to build a system that can continually understand and solve new tasks. labeled examples, as the mainstream task representation, are unlikely to be available in large numbers or even do not exist. then, is there any other task representation that can contribute to task comprehension? task instructions provide another dimension of supervision for expressing the task seman-tics. instructions often contain more abstract and comprehensive knowledge of the target task than individual labeled examples. with the availability of task instructions, systems can be quickly built to handle new tasks, especially when task-specific annotations are scarce @xcite . instruction following is inspired by the typical human learning for new tasks, e.g., a little kid can well solve a new mathematical task by learning from its instruction and a few examples. this new learning paradigm has recently begun to attract the attention of the machine learning and nlp communities. despite the importance, frontier research in instruction following is still struggling with the following questions. first, should instructions be constructed to express the target task as detailed as possible (e.g., mturk instructions @xcite ) or to align with the format of supervising tasks (e.g., natural language inference @xcite or language modeling (brown et al., 2020)) as well as possible? second, how to effectively encode instructions that may consist of some specific requirements such as "maximal output length 5", and "do not generate anything else apart from one of the following • • •"? third, what are the factors (e.g., model size, task numbers) that influence a system's generalization, robustness, etc.? fourth, how to evaluate instruction-following systems? last, what is the future for academia and industry in this chatgpt era? in this tutorial, we will systematically review several lines of frontier research on developing systems that are supervised by task instructions. beyond introducing pioneering work that parsed instructions to cope with individual tasks, such as soccer game @xcite , software control @xcite , etc., we will focus on recent llm-based approaches for crosstask generalization given task instructions. specifically, in light of the heterogeneous formats and dis-parate rationales underlying instructions, we shall endeavor to establish a unified lens for interpreting the essence of various instructions. subsequently, a structured exposition and critical analysis will be undertaken, encompassing a spectrum of aspects such as diverse instruction-following datasets, rigorous evaluation methodologies, multifaceted performance-influencing factors, and lingering concerns within this domain. participants will learn about recent trends and emerging challenges in this topic, representative tools and learning resources to obtain ready-to-use models, and how related technologies benefit enduser nlp applications. ## outline of tutorial content this half-day tutorial presents a systematic overview of recent advancements in nlp with supervision from task instructions. the detailed contents are outlined below. ## specification of the tutorial the proposed tutorial is considered a cutting-edge tutorial that introduces new frontiers in instruction-driven nlp. the presented topic has not been covered by any acl/emnlp/eacl/naacl/aacl/coling tutorials in the past 4 years. a tiny overlap exists between our section "lm-oriented task instructions" and the acl tutorial @xcite , which presented llm techniques for nlp. but @xcite focused on various training techniques, such as self-training, meta-training, etc., our tutorial has a broader scope of instruction learning, in which prompt-based llm is merely a sub-area. ## tutorial instructors the following are biographies of the speaker. wenpeng yin is an assistant professor in the department of computer science and engineering at penn state university. his research focuses on nlp with three sub-areas: (i) learning from task instructions; (ii) information extraction; (iii) nlp for education, bioinformatics, etc. dr. yin has presented the tutorial "indirectly supervised natural language processing" at acl'23, and tutorial "learning from task instructions" at kon-vens'23. additional information is available at www.wenpengyin.org . qinyuan ye is a fifth-year ph.d. student at the university of southern california, advised by prof. xiang ren. her research interest lies in natural language processing. in particular she is interested in approaches that reduce human annotation efforts, including methods leveraging distant supervision, high-level human supervision (e.g., explanations, instructions), and meta-learning. additional information is available at yeqy.xyz. pengfei liu is an associate professor at shanghai jiaotong university and leads the generative artificial intelligence research lab (gair). his research topics currently focus on information extraction, text generation, language pre-training, and nlp system evaluation. he won the best demo paper award in acl 2021 and the outstanding demo paper award in acl 2022. homepage: http://pfliu.com . xiang ren is an associate professor in computer science and the andrew and erna viterbi early career chair at usc. ren's research seeks to build generalizable nlp systems that can handle a wide variety of language tasks and situations. he works on new algorithms and datasets to make nlp systems cheaper to develop and maintain, arm machine models with common sense, and improve model's transparency and reliability to build user trust. his research work has received several best paper awards in top nlp and ai conference venues. ren has been awarded an nsf career award, multiple faculty research awards from google, facebook, amazon, jp morgan and sony, and the 2018 acm sigkdd doctoral dissertation award. he was named forbes' asia 30 under 30 in 2019. ren has presented a number of tutorials, such as knowledge-augmented methods for natural language processing at acl 2022, scalable construction and reasoning of massive knowledge bases at naacl 2018, and other related tutorials at www'18, cikm'17, etc. homepage: https://shanzhenren.github.io . hinrich schütze is chair of computational linguistics and co-director of the center of information and language processing at ludwig-maximilians-universität münchen (lmu munich), germany. he was the president of the association for computational linguistics in 2020, and general chair of acl 2013. in 2022, prof. schütze was elected as acl fellow. prior to joining lmu munich, he was a professor of theoretical computational linguistics at the university of stuttgart. hinrich holds a ph.d. in computational linguistics from stanford university. additional information is available at https://schuetze.cis.lmu. de .