Meta Learning and Its Applications to Natural Language Processing (2021)

Abstract: Deep learning based natural language processing (NLP) has become the mainstream of research in recent years and significantly outperforms conventional methods. However, deep learning models are notorious for being data and computation hungry. These downsides limit the application of such models from deployment to different domains, languages, countries, or styles, since collecting in-genre data and model training from scratch are costly. The long-tail nature of human language makes challenges even more significant. Meta-learning, or ‘Learning to Learn’, aims to learn better learning algorithms, including better parameter initialization, optimization strategy, network architecture, distance metrics, and beyond. Meta-learning has been shown to allow faster fine-tuning, converge to better performance, and achieve amazing results for few-shot learning in many applications. Meta-learning is one of the most important new techniques in machine learning in recent years. There is a related tutorial in ICML 2019 and a related course at Stanford, but most of the example applications given in these materials are about image processing. It is believed that meta-learning has great potential to be applied in NLP, and some works have been proposed with notable achievements in several relevant problems, e.g., relation extraction, machine translation, and dialogue generation and state tracking. However, it does not catch the same level of attention as in the image processing community. In the tutorial, we will first introduce Meta-learning approaches and the theory behind them, and then review the works of applying this technology to NLP problems. This tutorial intends to facilitate researchers in the NLP community to understand this new technology better and promote more research studies using this new technology.

## brief description deep learning based natural language processing (nlp) has become the mainstream of research in recent years and significantly outperforms conventional methods. however, deep learning models are notorious for being data and computation hungry. these downsides limit such models' application from deployment to different domains, languages, countries, or styles, since collecting in-genre data and model training from scratch are costly. the long-tail nature of human language makes challenges even more significant. meta-learning, or 'learning to learn', aims to learn better learning algorithms, including better parameter initialization, optimization strategy, network architecture, distance metrics, and beyond. meta-learning has been shown to allow faster finetuning, converge to better performance, and achieve outstanding results for few-shot learning in many applications. meta-learning is one of the most important new techniques in machine learning in recent years. there is a related tutorial in icml 2019 1 and a related course at stanford foot_1 , but most of the example applications given in these materials are about image processing. it is believed that metalearning has excellent potential to be applied in nlp, and some works have been proposed with notable achievements in several relevant problems, e.g., relation extraction, machine translation, and dialogue generation and state tracking. however, it does not catch the same level of attention as in the image processing community. in the tutorial, we will first introduce metalearning approaches and the theory behind them, and then review the works of applying this technology to nlp problems. ## type of the tutorial the type of tutorial is cutting-edge. meta-learning is a newly emerging topic. the area of natural language processing has seen a growing number of papers about meta-learning. however, there is no tutorial systematically reviewing relevant works at acl/emnlp/naacl/eacl/coling. ## tutorial structure and content a typical machine learning algorithm, e.g., deep learning, can be considered as a sophisticated function. the function takes training data as input and a trained model as output. today the learning algorithms are mostly human-designed. these algorithms have already achieved significant progress towards artificial intelligence, but still far from optimal. usually, these algorithms are designed for one specific task and need a lot of labeled training data. one possible method that could overcome these challenges is meta-learning, also known as 'learning to learn', which aims to learn the learning algorithm. in the image processing research community, meta-learning has shown to be successful, especially few-shot learning. it has recently also been widely adopted to a wide range of nlp applications, which usually suffer from data scarcity. this tutorial has two parts. in part i, we will introduce several meta-learning approaches (estimated 1.5 hours). in part ii, we will highlight the applications of the meta-learning methods to nlp (estimated 1.5 hours). ## diversity as the main applications of the meta-learning approaches are to find better metrics, model architec-tures, or initializations such that the meta-trained model can generalize well in new tasks with limited data, the approach is often used at efficient knowledge transferring between domains and languages, and has seen many promising results. meta-learning has the potential to democratize the progress of machine learning and nlp for different domains, languages, and countries in a scalable way. ## prerequisites for the attendees the attendees have to understand derivatives as found in introductory calculus and understand basic machine learning concepts such as classification, model optimization, and gradient descent. ## reading list we encourage the audience to read the papers of some well-known meta-learning techqnieus before the tutorial, which are listed below. hung-yi lee foot_2 received the m.s. and ph.d. degrees from national taiwan university (ntu), taipei, taiwan, in 2010 and 2012, respectively. from september 2012 to august 2013, he was a postdoctoral fellow in research center for information technology innovation, academia sinica. from september 2013 to july 2014, he was a visiting scientist at the spoken language systems group of mit computer science and artificial intelligence laboratory (csail). he is currently an associate professor of the department of electrical engineering of national taiwan university, with a joint appointment at the department of computer science information engineering of the university. his research focuses on machine learning (especially deep learning), speech processing and natural language processing. he owns a youtube channel teaching deep learning (in mandarin) with more than 5m views and 60k subscribers. ngoc thang vu foot_3 received his diploma (2009) and phd (2014) degrees in computer science from