Meta-Learning for Few-Shot Named Entity Recognition (2021)

Abstract: Meta-learning has recently been proposed to learn models and algorithms that can generalize from a handful of examples. However, applications to structured prediction and textual tasks pose challenges for meta-learning algorithms. In this paper, we apply two meta-learning algorithms, Prototypical Networks and Reptile, to few-shot Named Entity Recognition (NER), including a method for incorporating language model pre-training and Conditional Random Fields (CRF). We propose a task generation scheme for converting classical NER datasets into the few-shot setting, for both training and evaluation. Using three public datasets, we show these meta-learning algorithms outperform a reasonable fine-tuned BERT baseline. In addition, we propose a novel combination of Prototypical Networks and Reptile.

## introduction the usage of natural language understanding (nlu) technologies has spread widely in the last decade thanks to the recent jump in accuracy due to deep neural networks (dnn). in addition, dnn libraries have made easier than ever the productization of nlu technologies. applications have spread in quality and quantity with the broadened usage of chat bots by customer services, the development of virtual assistants (e.g. amazon alexa, google home, apple's siri or microsoft cortana) and the need of document parsing (e.g. medical reports, receipts, tweets, news articles) for data extraction. these applications often rely on ner to locate and classify named entities in text. ner aims at extracting named entities (e.g. "artist", "city" or "restaurant type") from a sequence of words. this problem is often approached @xcite as a sequence labeling task that assigns to each word one of the different entity types or the "other" label for words that do not belong to any named entity. the wide variety of applications has made the need for domain specific data the main bottleneck to train or fine-tune statistical models. this data is often acquired by running the application itself and collecting user inputs. then, the annotation effort can be significantly reduced using active learning @xcite or semi-supervised learning @xcite . however, to reach this bootstrapping stage, statistical models have to perform reasonably before being exposed to users. indeed, low performing models can turn away users or shift the input distribution as users lose engagement with features that do not work. transfer learning @xcite is an efficient way to cope with the data shortage by extracting task-agnostic high-level features. in particular, for ner, fine-tuning language models @xcite @xcite allows achieving state-of-the-art performances @xcite . however, fine tuning to specific tasks still requires a reasonable amount of data, especially for a task like ner with large structured label spaces. in certain cases, for example to learn personalized models or for products with restricted budgets, only a handful "reference" examples are available. as we will show, in such scenarios where very few training examples are available, transfer learning has its limitations. few-shot learning (fsl) is a rapidly growing field of research, reviewed in section 2, that aims at building models that can generalize from very few examples as detailed in @xcite . this area of research is motivated by the ability of humans and animals to learn object categories from few examples, and at a rapid pace. in particular, inductive bias @xcite has been identified for a long time as a key component to fast generalization to new inputs. previous work has suggested that meta-learning @xcite can help quickly acquire knowledge from few examples by learning an inductive bias from a distribution of similar tasks but with different categories. in this paper, we leverage recent progress made in transfer learning and meta-learning to address few-shot ner. first, we provide a novel definition of few-shot ner in section 3.1 where few-shot ner aims at building models to solve ner tasks given only a handful of labeled utterances per entity type. then, in section 3.2, we define a transfer learning baseline consisting in fine-tuning a pretrained language model (bert @xcite using only few examples. in addition, we introduce an extension of prototypical networks @xcite , a metric-based model, capable of handling structured prediction. in particular, we detail how it can be combined with conditional random fields (crf) @xcite . in section 3.3, we explain how such models can be trained using meta-learning. in addition, we introduce the application of an optimization-based algorithm to ner, reptile @xcite , capable of metalearning a better initialization model. we also propose a novel combination of prototypical networks and reptile that brings the best of both worlds, performance and the ability to handle a different number of classes between training and testing. finally, in section 3.4, we show how to generate diverse and realistic fsl tasks, corresponding to the bootstrapping phase of ner systems, from classical ner datasets either for meta-training or metatesting. in section 4, we conduct an extensive evaluation on three public datasets: snips @xcite , task oriented parsing (top @xcite and google schema-guided dialogue state tracking (dstc8 @xcite where we compare our three meta-learning approaches to the transfer learning baseline. source code and datasets will be made available online. meta-learning takes a different approach by trying to learn an inductive bias on a distribution of similar tasks that can be utilized to build models from very few examples. there are four common approaches. model-based meta-learning relies on a meta-model to update or predict the weights of a task specific model @xcite . generation-based meta-learning @xcite produces generative models able to quickly learn how to generate task specific examples, often in the feature space @xcite . the other two approaches are explained in detail below. metric-based meta-learning is similar to nearest neighbors algorithms. in particular, several metricbased meta-learning methods @xcite @xcite have been proposed for few-shot classification where an embedding space or a metric is meta-learned and used at test time to embed the few support examples of new categories and the queries. prediction is performed by comparing embedded queries and support examples. in many cases, the loss function is based on a distance between the supports and the queries. more advanced losses have been proposed in @xcite @xcite for example based on triplet, ranking and max-margin losses. one of the issues with approaches listed above is that the distance is the same for all categories. thus, fort (2017); @xcite have explored scaling the distance for new categories. optimization-based meta-learning explicitly meta-learns an update rule or weight initialization that enables fast learning during meta-testing. in @xcite , they use an lstm meta-learner trained to be an optimization algorithm. however, this approach incurs a high complexity. in @xcite , the authors explored with success using ordinary gradient descent in the learner and meta-learning the initialization weights. however, this algorithm named maml, requires to back propagate through gradient updates and so rely on second order derivatives which are expensive to compute. they also proposed an algorithm, fomaml, relying only on first order deriva-tives. this idea has been extended by @xcite to propose an algorithm, reptile, that does not need a training-test split for each task as explained in section 3.3. note that, @xcite gives an overview of many meta-learning algorithms and propose a set of benchmarks to evaluate them. finally, instead of just learning a model initialization, @xcite propose to learn a full-stack stochastic gradient descent (sgd), including update direction, and learning rate. few-shot learning on textual data has been explored recently, mostly for text classification tasks. @xcite propose to meta-learn a set of distances and learn a task-specific weighted combination of those. jiang et al. (2018) @xcite . ner has been addressed in several works. in @xcite the task of interest consists of recognizing one class of named entities, for tag set extension or domain transfer. in our work, we extend the n-way k-shot setting to structured prediction. @xcite propose a crf with coarse-grained transitions between abstract classes. in @xcite the authors propose a task sampling algorithm based on intents which can result in leakage between metatraining and meta-testing sets. in @xcite the authors don't use pre-trained language models. as we will show subsequently our work differs significantly from those. first, our task sampling method, that can generate a very large amount of tasks, is key to learn efficiently an inductive bias. second, we utilize pre-trained language models. third, using a fine-grained crf, amenable to meta-learning, our model can learn sequential de-pendencies between labels. fourth, we fine-tune our meta-learned prototypical network per task and even utilize optimization-based meta-learning to improve the fine-tuning. those contributions are central in achieving the best performance on few-shot ner as shown in section 4. ## conclusions in this paper, we have proposed a new definition of few-shot learning for ner, not relying a coarsegrain approach, like in @xcite , based on the intent to generate tasks. we have shown that, combining fine-tuning language models, crf, diverse task generation, optimization-based and metric-based meta-learning, can significantly and consistently outperform transfer learning on three datasets. also, our combination of prototypical network and reptile is quite robust to mismatches in the number of shots or ways between metatraining and meta-testing. thus, our approaches are effective to bootstrap nlu systems. for future works, one specificity of few-shot ner has not been properly addressed yet. although different in every tasks, the definition of the background class ("other") is partially shared between tasks. this assumption could be better leveraged in our approaches to transfer some of that knowledge across tasks instead of treating the background class as a different entity type in every tasks. another interesting direction to explore is few-shot integration, when we have to build a model that performs well on tasks made of entity types seen and unseen during meta-training. linguistics, pages 2872-2881, florence, italy. association for computational linguistics. kang min yoo, youhyun shin, and sang goo lee. 2018. data augmentation for spoken language understanding via joint variational generation. mo yu, xiaoxiao guo, jinfeng yi, shiyu chang, saloni potdar, yu cheng, gerald tesauro, haoyu wang, and bowen zhou. 2018. diverse few-shot text classification with multiple metrics. in proceedings of the north american chapter of the association for computational linguistics. ruixiang zhang, tong che, zoubin ghahramani, yoshua bengio, and yangqiu song. 2018. metagan: an adversarial approach to few-shot learning. in advances in neural information processing systems. zijian zhao, su zhu, and kai yu. 2019. data augmentation with atomic templates for spoken language understanding. arxiv preprint arxiv:1908.10770.