Recent Advances in Pre-trained Language Models: Why Do They Work and How Do They Work (2022)

Abstract: Pre-trained language models (PLMs) are language models that are pre-trained on large-scaled corpora in a self-supervised fashion. These PLMs have fundamentally changed the natural language processing community in the past few years. In this tutorial, we aim to provide a broad and comprehensive introduction from two perspectives: why those PLMs work, and how to use them in NLP tasks. The first part of the tutorial shows some insightful analysis on PLMs that partially explain their exceptional downstream performance. The second part first focuses on emerging pre-training methods that enable PLMs to perform diverse downstream tasks and then illustrates how one can apply those PLMs to downstream tasks under different circumstances. These circumstances include fine-tuning PLMs when under data scarcity, and using PLMs with parameter efficiency. We believe that attendees of different backgrounds would find this tutorial informative and useful.

## brief description deep learning-based natural language processing (nlp) has become mainstream research in recent years and has shown significant improvements over conventional methods. among all deep learning methods, fine-tuning a self-supervisedly pretrained language model (plm) on downstream tasks of interest has become the standard pipeline in nlp tasks. ever since elmo @xcite and bert @xcite were proposed in 2018, models fine-tuned from plms have dominated numerous leader-boards in various tasks including question answering, natural language understanding, natural language inference, machine translation, and sentence similarity. aside from applying plms on various downstream tasks, many have been delving into understanding the properties and characteristics of plms, including the linguistic knowledge encoded in the representations of plms, and the factual knowledge the plms acquire during pre-training. while it has been three years since plms were first proposed, there is no sign of decay in the research related to plms. there were two tutorials focusing selfsupervised learning/plms: a tutorial in naacl 2019 @xcite and one in aacl 2020 1 . however, given the ever-evolving nature of this realm, it is conceivable that there have been significant progress in the study of plms. specifically, compared with plms back in 2019, when they are mostly held by tech giants and used in scientific research, the plms nowadays have become more widely adopted in various real-world scenarios by users with different hardware infrastructures and amount of data, and thus posing problems that have never arisen before. substantial progress, including possible answers to the effectiveness of plms and new training 1 https://www.youtube.com/watch?@xmath0, have been made to allow plms better deployed in more realistic settings. hence, we see it necessary and timely to inform the nlp community about the recent advances in plms through a well-organized tutorial. this tutorial is divided into two parts: why do plms work and how do plms work. type of the tutorial the type of this tutorial is cutting-edge. we will cover the cuttingedge advances in plms which have been flourishing in the nlp community since 2020. no tutorial has systematically reviewed any topics that we aim to cover (as listed in ## tutorial structure and content pre-trained language models are language models that are pre-trained on large-scaled corpora in a self-supervised fashion. traditional self-supervised pre-training tasks mostly involve recovering a corrupted input sentence, or auto-regressive language modeling. after these plms are pre-trained, they can be fine-tuned on downstream tasks. conventionally, these fine-tuning protocol includes adding a linear layer on top of the plms and training the whole model on the downstream tasks, or formulating the downstream tasks as a sentence completion task and fine-tuning the downstream tasks in a seq2seq way. fine-tuning plms on downstream tasks often yield exceptional performance gain, which is why plms have become so popular. in the first part of the tutorial (estimated 40 mins), we will summarize some findings that partially explain why plms lead to exceptional downstream performance. some of these results have helped researchers to design better pre-training and fine-tuning methods. in the second part (estimated 2 hrs 20 mins), we will introduce recent progress in how to pre-train and fine-tune plms; the new techniques covered in this part have been shown to bring significant efficiency in terms of hardware resource, training data, and model parameters while achieving superb performance. ## diversity plms have shown promising results on different domains and have boosted the performance of lowresource languages on many tasks. the why part covered in this tutorial has the potential to help individuals of different groups to pre-train their own plms more efficiently. the how part covered in this tutorial specifically focuses on how to apply plms under different real-world scenarios with data scarcity and restricted model parameters, which will enable individuals of different groups to apply plms on the domains of interest in a more realistic setting. we see this tutorial to benefit diverse groups in the community. the tutorial instructors are also diverse: chuang is a phd student in the usa, and lee and chiang are researchers in taiwan. also, chuang and chiang are currently ph.d. students familiar with precise implementations, while lee is a senior researcher with ten years of experience in human language processing research. this diversity in members enables our team to provide a thorough and detailed yet comprehensive and unified view on plms. ## prerequisites for attendees we expect the attendees to have basic machine learning concepts such as gradient descent and model optimization. the attendees will need to have basic knowledge in linear algebra and calculus to understand some contents in block (b) in ## reading list we encourage attendees to read the following emblematic papers on plms and transformer model architectures: ## open access we will allow our slides and video recording of the tutorial published in the acl anthology. all the slides and videos used in the tutorial, along with the reading lists related with the tutorial, will be updated at this tutorial website 10 .