Zero- and Few-Shot NLP with Pretrained Language Models (2022)

Abstract: The ability to efficiently learn from little-to-no data is critical to applying NLP to tasks where data collection is costly or otherwise difficult. This is a challenging setting both academically and practically—particularly because training neutral models typically require large amount of labeled data. More recently, advances in pretraining on unlabelled data have brought up the potential of better zero-shot or few-shot learning (Devlin et al., 2019; Brown et al., 2020). In particular, over the past year, a great deal of research has been conducted to better learn from limited data using large-scale language models. In this tutorial, we aim at bringing interested NLP researchers up to speed about the recent and ongoing techniques for zero- and few-shot learning with pretrained language models. Additionally, our goal is to reveal new research opportunities to the audience, which will hopefully bring us closer to address existing challenges in this domain.

## introduction the ability to efficiently learn from little-to-no data is critical to applying nlp to tasks where data collection is costly or otherwise difficult. this is a challenging setting both academically and practically-particularly because training neutral models typically require large amount of labeled data. more recently, advances in pretraining on unlabelled data have brought up the potential of better zero-shot or few-shot learning @xcite . in particular, over the past year, a great deal of research has been conducted to better learn from limited data using large-scale language models. in this tutorial, we aim at bringing interested nlp researchers up to speed about the recent and ongoing techniques for zero-and few-shot learning with pretrained language models. additionally, our goal is to reveal new research opportunities to the audience, which will hopefully bring us closer to address existing challenges in this domain. the detailed content of the tutorial is described in section 2. the tutorial will start by motivating the challenge of learning from limited data, and providing an overview of historical few-shot nlp techniques. the tutorial will then start mainly focusing on recent few-shot learning methods using language models. it will cover methods from manual engineering, better inference algorithms to better tuning methods. we will then discuss the impact of different pretraining objectives, and meta-training strategies. lastly, we will survey the current landscape of evaluation benchmarks, and their limitations. we will conclude the tutorial by suggesting open questions, and providing coding examples and web-based demonstrations instructing attendees how to easily use these methods using public resources. ## tutorial content and outline this tutorial covers methods for zero-and few-shot learning with pretrained language models (lms). the tutorial will be 3 hours long. tutorial materials will be made available at: https://github.com/allenai/ acl2022-zerofewshot-tutorial. introduction -(10 minutes) we will start by motivating why zero-and few-shot learning are important. in many situations, labelled data may be costly or otherwise difficult to procure. language model finetuning, the predominant training paradigm in use today, exhibits poor performance in low-data regimes @xcite . furthermore, as lms continue to grow in size, so do the associated costs of training and storing separate weights for each downstream task. recent work on zero-and few-shot learning with pretrained language models can provide a potential solution. earlier work -(15 minutes) in the second section, we will review well-established methods for zero-and few-shot learning that do not necessarily use lms, including data augmentation, semisupervised learning, consistency training and cotraining @xcite @xcite . language models as few-shot learners -(20 minutes) in the third section, we will focus on fewshot approaches using lms without any tuning. the fundamental observation in this section is that, by reformulating tasks as complete-the-sentence problems and potentially including training examples in-context, large pretrained language models can be used to solve nlp tasks without having to resort to finetuning. we will survey a few key papers, notably @xcite , and follow up work demonstrating the limitations of incontext learning @xcite . we will also discuss alternative approaches for calibrating and scoring lm outputs @xcite @xcite . prompt-based finetuning -(25 minutes) in the next section, we will discuss prompt-based finetuning, which relaxes the restriction that the lm weights cannot be updated. we will introduce the technique of pattern exploiting training @xcite which utilizes manually written cloze style prompts in conjunction with language model finetuning to attain higher accuracy and improved stability over the finetuning approach proposed by @xcite . we will then discuss a variety of related works that seek to streamline pet @xcite . in particular we will cover methods that try to automate the task of prompt-construction, either in the vocabulary space @xcite , or the embedding space (li and liang, 2021; @xcite @xcite . we will contrast these methods with non-tuning methods covered in the previous section, in terms of their performance, memory and computation requirement, amount of required engineering, and more. pretraining -(20 minutes) the following section will focus on the factor underlying the success of these methods-language model pretraining. first, we will provide a review of popular language model pretraining objectives and architectures. topics will include: causal @xcite vs. masked @xcite pretraining, encoder-only @xcite vs. decoder-only @xcite vs. encoderdecoder architectures @xcite , and the impact of training data @xcite @xcite . meta-training -(25 minutes) next we will discuss meta-training approaches that train the lm to adapt to zero-and few-shot use cases. a variety of work has demonstrated that transfer learning is extremely effective when trained on a diverse set of tasks and prompts @xcite . furthermore, recent papers propose to learn from instructions where the model is given instructions that humans would often read when performing a new task, e.g., in a crowdsourcing task @xcite . evaluation benchmarks -(25 minutes) we will then discuss few-shot evaluation benchmarks such as flex @xcite , fewnlu @xcite , the big-bench (big-bench collaboration, 2021) and crossfit @xcite ). we will discuss the problems in existing evaluations and how new few-shot evaluation benchmarks were carefully designed to measure a variety of scopes in generalization. we will also cover benchmarks specifically for instruction learning @xcite . ## type of the tutorial this tutorial will cover cutting-edge research in zero-and few-shot learning with pretrained language models. this topic has not been previously covered in *cl tutorials. ## breadth the tutorial covers a diverse set of topics related to zero-and few-shot learning including pretraining, prompting, finetuning, evaluation, open research questions, etc. the tutorial also briefly discusses pre-language models work but not in depth. note that most of the work we will cover is not authored by the presenters. ## diversity considerations the methods and techniques we are going to present are language-agnostic and can be easily applied to non-english data and tasks. zero-and few-shot learning can be relevant for low-resource languages and tasks (assuming there exist unlabeled resources to build a pretrained model). the tutorial covers work from diverse groups, both geographically (america, europe, asia) and gender. for instructors, three are senior and two are junior nlp researchers, one is female, and they represent two universities and one industry research lab. ## prerequisites we assume attendees are familiar with: ## reading list reading the following papers is nice to have but not required for attendance. in alphabetical order, iz beltagy iz beltagy is a research scientist at ai2 focusing on language modeling, transfer learning, summarization, explainability and efficiency. his research has been recognized with a best paper honorary mention at acl 2020 and an outstanding paper award at akbc 2021. he was a coinstructor of the tutorial on "beyond paragraphs: nlp for long sequences" (naacl-hlt 2021). he worked as a teaching assistant at the university of texas at austin teaching computer science. email: beltagy@allenai.org homepage: beltagy.net arman cohan arman cohan is a research scientist at ai2 and an affiliate assistant professor at university of washington, focusing on representation learning and transfer learning methods, as well as nlp applications in specialized domains and scientific text. his research has been recognized with a best paper award at emnlp 2017, an honorable mention at coling 2018, and harold n. glassman distinguished doctoral dissertation award in 2019. he was a co-instructor of the tutorial on "beyond paragraphs: nlp for long sequences" (naacl-hlt 2021). email: armanc@allenai.org homepage: armancohan.com robert l. logan iv robert l. logan iv is a ph.d. student at the university of california, irvine, advised by sameer singh and padhraic smyth. his research focuses on problems at the intersection of information extraction and language modeling, and encompasses recently published work on language model prompting that is relevant to this proposal. he has presented invited talks at the socal nlp symposium (2019), the chase-ci workshop (2019), and the uci center for machine learning seminar (2021). email: rlogan@uci.edu homepage: rloganiv.github.io sewon min sewon min is a ph.d. student in the paul g. allen school of computer science & engineering at the university of washington, advised by hannaneh hajishirzi and luke zettlemoyer. her research focuses on natural language understanding, question answering, and knowledge representation. she was a co-instructor of the tutorial on "beyond paragraphs: nlp for long sequences" (naacl-hlt 2021), and was a co-organizer of the 3rd workshop on machine reading for question answering (emnlp 2021), competition on efficient open-domain question answering (neurips 2020), and workshop on structured and unstructured kbs (akbc 2020, 2021). email: sewon@cs.washington.edu homepage: shmsw25.github.io sameer singh sameer singh is an associate professor of computer science at the university of california, irvine and an allen ai fellow at the allen institute for ai. he is working on large-scale and interpretable machine learning models for nlp. his work has received paper awards at acl 2020, akbc 2020, emnlp 2019, acl 2018, and kdd 2016. sameer has presented a number of tutorials, many relevant to this proposal, such as deep adversarial learning tutorial at naacl 2019, mining knowledge graphs from text tutorial at wsdm 2018 and aaai 2017, tutorial on interpretability and explanations in neurips 2020 and emnlp 2020, and tutorial on robustness in nlp at emnlp 2021. sameer has also received teaching awards at uci. email: sameer@uci.edu homepage: http://sameersingh.org/ ## ethical statement this tutorial covers work that extensively uses large (up to hundreds of billions of parameters) language models, which are associated with substantial financial and environmental costs @xcite , as well as other harms @xcite .